The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

Xunzhe Zhou; Yiyang Cai; Fengyi Wang; Ran Ju; Hanxiang Ren; Ruizhe Liu; Yu Zhang; Qian Luo

cs.ROAug 23, 2026

The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

Authors: Xunzhe Zhou, Yiyang Cai, Fengyi Wang, Ran Ju, Hanxiang Ren, Ruizhe Liu, Yu Zhang, Qian Luo, +4 more

Organizations: The University of Hong Kong · TranscEngram · Fudan University · Zhejiang University

Abstract

Humans imitate at the level of intent: given a demonstration, we infer its goal and carry it out with whatever tools, objects, and layouts are at hand. Current robot policies instead learn observation-to-action mappings from visual inputs and language instructions, without explicitly inferring the demonstrated task. Learning from human video thus remains largely trajectory-level: models can replay motions in near-identical scenes, but still struggle to imitate what the demonstrator intends rather than merely what they do. We introduce The Imitator Game, a four-level benchmark (L0-L3) that progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice and task understanding becomes necessary. We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date and the only one instantiated across all four levels in both real and simulated settings (20,000+ paired episodes, 50+ tasks, 6 domains), and Imitator Arena, an open platform for blind A/B human evaluation. Across nine state-of-the-art models, performance is stable from L0 to L2 but collapses at L3, identifying functional substitution - achieving the same intent through a different object affordance - as the decisive barrier to intent-level imitation. Human-video-conditioned models outperform caption-conditioned ones, yet every model falls below 13% zero-shot success on unseen tasks; fine-tuning IG-10K-pretrained models with only $10$ paired human-robot demonstrations yields large gains that grow with pretraining scale. The project website and access to Imitator Arena are available at https://imitator-game.github.io.

Figures & tables

Figure 1: The Imitator Game. (Left) The Imitator Game setup: a human demonstrates a manipulation task in the Demonstrator Scene (green), and a robot watches the video and reproduces the underlying intent in the Imitator Scene (orange) using whatever objects are available. (Right) The Imitator Arena evaluation platform: human evaluators can make structured blind A/B comparisons of how two anonymized model rollouts imitate against the demonstrator video.

Figure 2: Four levels of imitation, illustrated on a single demonstration: a human making tea by steeping a tea bag (leftmost filmstrip). Left to right, the imitator’s scene diverges further from the demonstration, so less and less of the demonstration can still be copied exactly. L0: same objects and layout; the imitator steeps the tea bag on the right, reproducing the demonstrated motion. L1: the layout is rearranged (tea bag in front rather than on the right), so the same final state must be reached by a different trajectory. L2: the tea bag is replaced by loose tea leaves of the same kind, so only the semantic task “make tea” is reproducible. L3: no tea is present, so the imitator serves the same intent with the available matcha powder, mixing it rather than steeping. Abstraction increases left to right.

Level	$P$	$A$ or $G$	$S$	Replication fidelity expected of the imitator
L0 (trajectory)	✓	✓	✓	Trajectory match: similar motion to $V$ .
L1 (object end-state)	$\times$	✓	✓	Same final object states as $V$ ; trajectory may differ.
L2 (semantic task)	n/a	$\times$	✓	Same task semantics as $V$ ; final states differ.
L3 (affordance-adapted)	n/a	$\times$	$\times$	Same underlying intent, via re-purposed affordances.

Table 1: The four levels of the imitator game as preservation patterns across three scene-property columns. ✓ = preserved between

\mathcal{S}_{\text{demo}}

and

\mathcal{S}_{\text{imit}}

;

\times

= changed; n/a = not applicable, as the object set has changed and configuration has no identity mapping. The right column states the highest fidelity the imitator’s scene permits.

Figure 3: The IG-10K dataset. (a) IG-10K contains $20{,}000+$ paired episodes: 11.7K real human-robot paired episodes, each with one human clip and one corresponding real-robot clip, plus 10K paired simulation episodes. These span household, supermarket, restaurant, logistics, hospital, and laboratory manipulation tasks. The montage, donut chart, and word cloud summarize the dataset scale and task diversity. (b) The object gallery shows the real and simulated assets used to instantiate the L0–L3 scene variations. (c) Each episode is released with dense offline annotations, including semantic masks, hand poses, depth images, and language descriptions.

Dataset	Human clips	Robot clips	Tasks	Depth	Env. aligned	Sim. env.	Multi view	Hand pose	Lang. ann.	Seg. mask
MIME [ 34 ]	8.3k	8.3k	20 task families	✓	✓	$\times$	✓	$\times$	$\times$	$\times$
BC-Z [ 14 ]	18.7k	25.9k	100+	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$
Vid2Robot [ 13 ]	10k	100k	Unknown	$\times$	$\Delta$	$\times$	$\times$	$\times$	$\times$	$\times$
RH20T [ 10 ]	110k	110k	147	✓	$\times$	$\times$	✓	$\times$	✓	$\times$
H&R [ 45 ]	2.6k	2.6k	10	$\times$	✓	$\times$	$\times$	$\times$	$\times$	$\times$
DexWild [ 37 ]	9.3k	1.4k	5 task families	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$

Table 2: IG-10K in context vs. existing human-robot paired manipulation datasets. ✓/

\times

= present/absent;

\Delta

= partial. Env. aligned : human and robot demos share the same or explicitly corresponding task scene, objects, and setting (not necessarily frame-level timing). Sim. env. : a paired, interactive simulation environment for the human demos (robot-rendered videos or offline trajectories do not count).

Figure 4: The experiment. Evaluation spans training tasks (left) and held-out test tasks (right). For each task, the first image shows the human reference video, followed by representative imitator trajectories under increasing scene mismatch from L0 to L3. The real-world evaluation covers seen-task imitation, unseen zero-shot transfer, few-shot adaptation, and Arena human judgments.

		Seen		ZS		Scr.		P+FT		$\Delta$
	Model	SR	Sub	SR	Sub	SR	Sub	SR	Sub	SR	Sub
VLA	OpenVLA [ 19 ]	0.29	0.60	0.06	0.20	0.14	0.34	0.20	0.44	+0.06	+0.10
	RDT-1B [ 24 ]	0.43	0.72	0.07	0.19	0.35	0.55	0.21	0.43	–0.14	–0.11
	GR00T-N1.6 [ 2 ]	0.51	0.81	0.03	0.17	0.21	0.49	0.67	0.79	+0.46	+0.31
	$\pi_{0.5}$ [ 12 ]	0.73	0.89	0.09	0.22	0.80	0.86	0.85	0.91	+0.05	+0.04
Skill	UniSkill [ 17 ]	0.75	0.89	0.07	0.17	0.27	0.46	0.59	0.76	+0.32	+0.30

Table 3: Simulation results. Seen, ZS, and P+FT are averaged over the

\{15,30,45\}

task scales. Seen = the five training tasks; ZS = zero-shot evaluation on unseen tasks; Scr. = trained from scratch on

10

demonstrations of the unseen task, without IG-10K pre-training; P+FT = IG-10K pre-trained and then fine-tuned on the same

10

demonstrations;

\Delta=\text{P+FT}-\text{Scr.}

; Sub = sub-goal success rate.

	Seen			ZS			Scr.			P+FT			$\Delta$
Model	SR	$\overline{Q}$ $\uparrow$	WR $\uparrow$	SR	$\overline{Q}$ $\uparrow$	WR $\uparrow$	SR	$\overline{Q}$ $\uparrow$	WR $\uparrow$	SR	$\overline{Q}$ $\uparrow$	WR $\uparrow$	SR	$\overline{Q}$ $\uparrow$	WR $\uparrow$
$\pi_{0.5}$ [ 12 ]	0.51	6.87	0.56	0.04	2.34	0.04	0.27	5.52	0.50	0.36	5.89	0.73	+0.09	+0.37	+0.23
XSkill [ 46 ]	0.63	7.49	0.89	0.29	5.24	0.50	0.28	5.40	0.52	0.49	6.72	0.95	+0.21	+1.32	+0.43
ACT/DINOv2 [ 48 , 29 ]	0.49	6.70	0.41	0.26	5.44	0.42	0.22	5.38	0.24	0.35	5.88	0.70	+0.13	+0.50	+0.46
DP/DINOv2 [ 8 ]	0.39	6.19	0.14	0.22	4.96	0.25	0.22	5.32	0.24	0.31	5.77	0.58	+0.09	+0.45	+0.34

Table 4: Real-world Arena results combining seen performance and unseen transfer. Seen, ZS, and P+FT are averaged over the

\{15,30,45\}

task scales. Settings as in Table 3 .

\overline{Q}

= mean human imitation score on the

0

–

10

rubric of Appendix B.3 ; WR = win rate in blind A/B comparisons. On hardware there is no automated metric, so SR here is the human success judgment.

	L0		L1		L2		L3
Model	SR	$\overline{Q}$ $\uparrow$	SR	$\overline{Q}$ $\uparrow$	SR	$\overline{Q}$ $\uparrow$	SR	$\overline{Q}$ $\uparrow$
$\pi_{0.5}$ [ 12 ]	0.44	6.27	0.35	5.77	0.37	5.97	0.28	5.57
XSkill [ 46 ]	0.53	6.80	0.57	7.20	0.56	7.23	0.29	5.63
ACT/DINOv2 [ 48 , 29 ]	0.37	6.00	0.37	5.90	0.33	5.77	0.33	5.83
DP/DINOv2 [ 8 ]	0.33	5.97	0.37	6.00	0.29	5.70	0.25	5.43
Average	0.42	6.26	0.42	6.22	0.39	6.17	0.29	5.62

Table 5: Real-world P+FT success by hierarchy level.

\mathrm{SR}

and human imitation score

\overline{Q}

, computed from real-world pretrain+finetune trials and averaged over the

\{15,30,45\}

-task scales. The corresponding line plot is Appendix Figure 12 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: IG-10K data collection. (left) The real-world data are collected with VR teleoperation on a dual-arm robot platform, with visual data recorded by three third-view RealSense cameras and one front-facing ZED2i RGB camera. (right) simulation data are collected with handcrafted motion-planning waypoints and trajectories planned using OMPL.

Domain	Task	L0	L1	L2	L3	Notes
Household (12)	H1: Stir Soup in Bowl	✓	✓	✓	✓
	H2: Return Book to Shelf	✓	✓	✓	✓
	H3: Fold Clothes into Basket	✓	✓	✓	✓
	H4: Return Remote to Box	✓	✓	✓	✓
	H5: Put Magazine in Folder	✓	✓	✓	✓
	H6: Return Body Wash to Holder	✓	✓	✓	✓

Appendix

Table 6: The IG-10K task suite. All base tasks by domain, with the levels at which each is instantiated (✓ present, – omitted). Per-domain counts: Household 12, Supermarket 7, Restaurant 17, Logistics 4, Hospital 6, Laboratory 7.

Corpus	Tasks
$\mathcal{C}_{15}$	StirSpoon , PlaceClothBasket, PlaceMagazineFolder, PickWash, PlaceChipsRack, PlaceFruitBox, PlacePlateRack , CutFruit, PlaceFileFolder , PlaceBrushRest, CleanCup, GrindFood, LiftLidFromSkillet, FoldTowel , PlaceMugRack
$\mathcal{C}_{30}$ (adds)	PlaceCommodityRack, PourKetchupFries, WipePot, CleanDesk, TransFood, PickTennisBallGolfBall, PickPillToRegions, PourLiquidCup, PlaceCupPlate, PutCubeOnScale, PourCup, PutBox, KnifeBowlFork, PressJuicer, PlaceScrewdriver
$\mathcal{C}_{45}$ (adds)	PlaceBookBookcase, PickAppleBasket, PickAppleBananaToBaskets, PickAppleToScale, PickFruitsToPlate, PlaceFoodScale, PressStapler, ScanPillBottle, PlaceShoeBox, OpenBox, PlacePillBox, PourLiquidMug, OpenLiquidCap, PourLiquidFilter, PlaceBurgerTray
Unseen	PickRemoteControl, ScanMilkBox, PourKettle, PickFood, FoldBox

Appendix

Table 7: Pre-training corpora and the unseen split. The three corpora are nested; the

30

-task corpus adds the listed tasks to the

15

-task corpus, and the

45

-task corpus adds its own to the

30

-task corpus. Bold marks the five seen evaluation tasks. The five unseen tasks appear in no corpus.

Figure 6: The Imitator Arena. The web interface shows the task description and human reference video on the left, and two anonymized model rollouts from the same imitator scene. Human evaluators judge each rollout for task completion (SUCCESS/PARTIAL/FAILURE), assign an imitation-quality score on a fixed 0–10 scale, and make an overall pairwise preference or tie decision. Level labels and model identities are hidden to focus evaluation on imitation of the demonstrated intent.

Score	Anchor
$0$	Complete failure
$2$	Minimal intent understanding
$5$	Partial imitation, no adaptation
$8$	Good imitation with minor errors
$10$	Perfect imitation

Appendix

Table 8: Imitation-quality rubric. The five fixed anchors shown to human evaluators for the per-rollout score

q\in[0,10]

(question (2)). Intermediate integers interpolate between adjacent anchors.

Figure 7: Adapting three paradigms to a unified control interface $a_{t}=\pi(V,o_{t})$ . Across all families the backbone is frozen and only the action head is trained on IG-10K. (a) VLA + text: a fixed captioner $T$ converts $V$ to a language prompt for a frozen vision–language backbone. (b) Skill-based: a frozen encoder extracts a cross-embodiment skill representation from $V$ , and a skill-conditioned policy acts on the retrieved skill index. (c) Video encoder + VA: a frozen encoder maps $V$ to a task embedding concatenated with the observation tokens for ACT, Diffusion Policy, or VQ-BeT.

Figure 8: Q1: which imitation interface is strongest. (a) Each trained variant on the

(\mathrm{seen\text{-}SR},\,\mathrm{P{+}FT\text{-}SR})

plane, automated channel; the video families cluster upper-right while the VLA family stretches along the diagonal. (b) Distribution of seen-task

\mathrm{SR}

(automated) within each paradigm; the within-family spread exceeds the widest between-family gap for VLA and Video-VA, and Video-Skill is the tightest. (c) Simulation Arena, human channel. Bar length is the blind A/B win rate

\mathrm{WR}

; the label beside each bar gives that variant’s

\mathrm{WR}

together with its mean imitation score

\overline{Q}

, which is a separate quantity and is not plotted. The human channel reproduces the ranking that the automated channel gives in panels (a,b) on the same rollouts.

Figure 9: Q1: the frozen video encoder. Success rate for the three Video-VA action heads crossed with the three frozen encoders; the two simulation panels are the automated channel and the real-world panel is the Arena channel. Video encoders run with ACT on real-world hardware were only at the

45

-task pre-training scale.

Figure 10: Q2: scaling the paired pre-training corpus. (a) P+FT

\mathrm{SR}

vs. corpus size, simulation (solid, automated) and real world (dashed, Arena); the two grey horizontal lines are the scratch mean for each domain – simulation over all

15

variants, real world over the four representative models – computed on the same basis as the Scr. columns of Table 3 and Table 4 . (b) Per-model

\Delta\mathrm{SR}

from

15

45

pre-training tasks; real-world models are hatched and are Arena numbers. Positive for

18

19

variants. (c) Real-world unseen zero-shot (Arena) vs. scale: the three video-conditioned models improve monotonically while the caption-conditioned

\pi_{0.5}

stays at the floor.

Figure 11: Q2: scaling resolved by hierarchy level. Unseen-task success against the number of pre-training tasks, separately per level, for (a) simulation zero-shot, (b) simulation P+FT, (c) real-world zero-shot and (d) real-world P+FT. Simulation curves average all

15

trained variants; real-world curves average the four representative models. Note the different

y

ranges for each columns.

	Sim ZS (auto.)			Sim P+FT (auto.)			Real ZS (Arena)			Real P+FT (Arena)
Level	15	30	45	15	30	45	15	30	45	15	30	45
L0	0.08	0.06	0.06	0.52	0.59	0.62	0.18	0.23	0.23	0.34	0.44	0.48
L1	0.14	0.09	0.07	0.48	0.59	0.58	0.18	0.20	0.24	0.35	0.45	0.45
L2	0.04	0.07	0.06	0.34	0.36	0.41	0.14	0.16	0.20	0.31	0.37	0.49
L3	0.03	0.01	0.04	0.47	0.55	0.59	0.19	0.23	0.25	0.23	0.28	0.36

Appendix

Table 9: Success rate by hierarchy level and pre-training scale. Unseen-task zero-shot (ZS) and pre-train

+

fine-tune (P+FT) success, resolved by level and by the number of pre-training tasks (

15/30/45

). Simulation numbers average all

15

trained variants; real-world numbers average the four representative models. Simulation columns are the automated channel, real-world columns the Arena channel. These are the numbers plotted in Figure 11 .

Figure 12: Q3: per-level success and completion quality. (a) Real-world seen-task

\mathrm{SR}

(Arena, left axis, solid) across L0–L3, one curve per representative model; each model’s imitation score

\overline{Q}

(right axis, dashed, same color, also Arena) is overlaid. (b) Real-world P+FT

\mathrm{SR}

and

\overline{Q}

(Arena), same layout as (a). (c) Simulation seen-task

\mathrm{SR}

(automated, left axis, solid) by paradigm family; each family’s Sub-SR

\times 10

(right axis, dashed, same color, also automated) is overlaid. (d) Simulation unseen-task P+FT

\mathrm{SR}

and Sub-SR

\times 10

(automated), same layout as (c).

	ACT/DINOv2					DP/DINOv2
Demo video	L0	L1	L2	L3	Avg	L0	L1	L2	L3	Avg
original	0.87	0.77	0.72	0.88	0.81	0.72	0.66	0.59	0.71	0.67
similar	0.32	0.29	0.32	0.32	0.31	0.02	0.05	0.01	0.04	0.03
unrelated	0.34	0.28	0.39	0.19	0.30	0.00	0.00	0.00	0.00	0.00
	$\pi_{0.5}$					XSkill
Demo video	L0	L1	L2	L3	Avg	L0	L1	L2	L3	Avg

Appendix

Table 10: Demonstration-swap sanity check (simulation, seen tasks,

45

-task checkpoints, automated channel). Success rate when the conditioning demonstration is the correct one ( original ) versus a video of a similar or an unrelated task, with the imitator scene unchanged.

Figure 13: Validity of the evaluation channels. (a) Automated vs. Arena scoring of the identical simulation rollouts (

x

: automated

\mathrm{SR}

;

y

: Arena

\mathrm{SR}_{\mathrm{human}}

). Dashed line is

y=x

. (b) The same two channels, resolved by hierarchy level on the seen-task rollouts (left axis: automated

\mathrm{SR}

and Arena

\mathrm{SR}_{\mathrm{human}}

; right axis: automated Sub-SR

\times 10

and Arena

\overline{Q}


Task	DP				ACT				Mean
	L0	L1	L2	L3	L0	L1	L2	L3	DP	ACT
CleanCup	40	50	20	100	30	30	70	90	52.5	55.0
CleanDesk	10	60	40	60	80	100	100	100	42.5	95.0
CutFruit	100	50	0	100	100	100	0	100	62.5	75.0
FoldBox	100	100	100	100	100	100	100	100	100.0	100.0

Appendix

Table 11: Per-task reference results for simulation success rate. Each cell reports success rate (%) over 10 evaluation episodes for the corresponding task and imitation level.

Explore similar work

Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
Feb 22, 2026Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang +3Learning from Demonstration Robot Imitation Learning
School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.
Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning
Aug 12, 2025Ben Zandonati, Tomás Lozano-Pérez, Leslie Pack KaelblingRobot Skill Learning Generalization in Robotic Manipulation
MIT CSAIL
Demo-JEPA: Joint-Embedding Predictive Architecture for One-shot Cross-Embodiment Imitation
May 20, 2026Jingyang He, Guangrun Li, Jieyu Zhang +3Robot Skill Learning World Models for Robotics
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · University of Washington

Feb 22, 2026J/K move · Enter open · S save

Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation

Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang, Ziyan Gao+2

School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.

Learning from Demonstration Robot Imitation Learning Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Learning from Demonstration (LfD) offers a promising paradigm for robot skill acquisition. Recent approaches attempt to extract manipulation commands directly from video demonstrations, yet face two critical challenges: (1) general video captioning models prioritize global scene features over task-relevant objects, producing descriptions unsuitable for precise robotic execution, and (2) end-to-end architectures coupling visual understanding with policy learning require extensive paired datasets and struggle to generalize across objects and scenarios. To address these limitations, we propose a novel ``Human-to-Robot'' imitation learning pipeline that enables robots to acquire manipulation skills directly from unstructured video demonstrations, inspired by the human ability to learn by watching and imitating. Our key innovation is a modular framework that decouples the learning process into two distinct stages: (1) Video Understanding, which combines Temporal Shift Modules (TSM) with Vision-Language Models (VLMs) to extract actions and identify interacted objects, and (2) Robot Imitation, which employs TD3-based deep reinforcement learning to execute the demonstrated manipulations. We validated our approach in PyBullet simulation environments with a UR5e manipulator and in a real-world experiment with a UF850 manipulator across four fundamental actions: reach, pick, move, and put. For video understanding, our method achieves 89.97% action classification accuracy and BLEU-4 scores of 0.351 on standard objects and 0.265 on novel objects, representing improvements of 76.4% and 128.4% over the best baseline, respectively. For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations. The project website is available at https://thanhnguyencanh.github.io/LfD4hri.

Abstract

Explore similar work

Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation

Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning

Demo-JEPA: Joint-Embedding Predictive Architecture for One-shot Cross-Embodiment Imitation