ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use

Mohamed Elmoghany; Kehan Li; Ruoqu Chen; Tarun Chiruvolu; Shutong Zhang; Ryan Wang; Stephen Tian; Yaru Niu

cs.RONov 6, 2026

ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use

Authors: Mohamed Elmoghany, Kehan Li, Ruoqu Chen, Tarun Chiruvolu, Shutong Zhang, Ryan Wang, Stephen Tian, Yaru Niu,

Organizations: Stanford University · Carnegie Mellon University

Abstract

Creative tool use is a fundamental human capability and an important step toward general-purpose embodied intelligence. It requires agents to reason about object affordances, physical constraints, and manipulation strategies, especially when standard tools are unavailable. However, progress in this area is limited by the lack of real-world datasets that capture human tool-use behavior with sufficient scale, fidelity, and diversity. Existing benchmarks are often simulation-based, focus on simplified interactions, or lack detailed observations and annotations. We introduce ToolDex, a large-scale egocentric-exocentric (ego-exo) human demonstration dataset for creative physical tool use. ToolDex captures scenarios in which humans select, adapt, and repurpose available objects as tools. Using synchronized egocentric cameras and eight exocentric cameras, the dataset provides high-fidelity multi-view observations and rich annotations for hands, tools, target objects, and their deformable interactions. Beyond data collection, we establish a benchmark suite characterizing three core capabilities: physical tool-use reasoning, sparse-view 3D and 4D reconstruction, and robot policy learning. ToolDex provides a foundation for studying human tool-use behavior and developing embodied agents that can adaptively use tools in complex real-world environments. Dataset visualizations and benchmarking results are available at https://tooluse-dataset.github.io/.

Figures & tables

Figure 1: ToolDex Overview. The dataset captures creative physical tool-use behaviors in everyday tabletop scenarios, where humans repurpose available objects as tools for manipulation (left). It explicitly models full hand–tool–target-object interactions, including for challenging deformable targets (middle). Using synchronized ego-exo multi-view capture, ToolDex provides rich annotations across views, including hand pose, tool, and target-object segmentation (right).

Dataset	#Tools	#Target Obj.	#Actions	#Frames	#Trials	Ego/Exo	Channels	Hand Pose	Obj. Pose	Seg. Mask
TACO Liu et al. (2024)	20	–	15	5.2M	2.5K	1 / 12	RGB (Ego RGB-D)	✓	✓	✓
OakInk2 Zhan et al. (2024)	75	–	60 primitives	4.01M	2.8K	1 / 3	RGB	✓	✓	✓
HOT3D Banerjee et al. (2025)	33	–	–	3.7M	2.8K	2–3 / 0	RGB + mono	✓	✓	✓
GigaHands Fu et al. (2025)	417	–	1467	183M	13.9K	0 / 51	RGB	✓	✓	✓
TOOLDEX	45	23	16	58.3M	10.8K	1 / 8	RGB-D	✓	✓	✓

Table 1: Comparison of ToolDex with recent hand–object-interaction datasets. Only ToolDex explicitly separates tool and target object roles (Section 3.1 ). For prior datasets the dash in #Target Obj. indicates a single combined object inventory rather than the absence of targets.

Figure 2: Combinatorial coverage of ToolDex . Binary heatmaps of the (tool, action, target) combinations in the dataset, with each axis sorted from most to least frequent.

Figure 3: Data collection setup. Left: The diverse tool objects in ToolDex . Middle: The multi-camera rig, consisting of eight Intel RealSense RGB-D cameras and an Apple Vision Pro headset with a mounted ZED Mini stereo camera. Right: Three sampled camera views, including two exocentric and one egocentric view.

Figure 4: 3DGS and 4DGS prediction for camera view withheld during training and prediction of novel views using InstantSplat + Difix3D. Additional visualizations are available on our website.

	Static (3DGS)			Dynamic (4DGS)
	SplatFormer	IS	IS+Difix3D	IS-4D	IS-4D+Difix3D+GSFix3D
PSNR $\uparrow$	13.09	19.97	20.45	19.60	15.97
MUSIQ $\uparrow$	33.11	50.66	55.11	51.00	69.92

Table 2: Novel-view synthesis results for data captured by 8 sparse cameras, for both static (3DGS) and dynamic (4DGS) settings. PSNR is computed against a held-out camera view (cam 6 ); MUSIQ scores are computed after training with all 8 cameras (no ground truth required). Methods: SplatFormer ( Chen et al., 2025 ) , InstantSplat (IS) ( Fan et al., 2024 ) , Difix3D ( Wu et al., 2025 ) , GSFix3D ( Yin et al., 2026 ) ; MUSIQ ( Ke et al., 2021 ) .

Model	Tool Acc. $\uparrow$	BBox IoU $\uparrow$
GPT-5-mini	.662	.300
GPT-6-Astra	.800	.891
Claude-Sonnet-4.6	.800	.641
Claude-Opus-4.7	.800	.564
Gemini-3-Flash	.756	.050
Gemini-3.1-Pro	.781	.010

Table 3: Benchmark results on tool selection accuracy and bounding box IoU. Models: GPT-5-mini ( OpenAI, 2025 ) , GPT-6-Astra ( OpenAI, 2026 ) , Claude-Sonnet-4.6 ( Anthropic, 2026 ) ,Claude-Opus-4.7 ( Anthropic, 2026 ) Gemini-3.1-Pro ( Google DeepMind, 2026 ) , Qwen-3.6-27B ( Qwen Team, 2026 )

Model	Eval. success $\uparrow$	Tool-center		Corr.-aware
		Mean L2 $\downarrow$	Chamfer-L2 $\downarrow$	Mean L2 $\downarrow$	Chamfer-L2 $\downarrow$
GPT-6 Astra.	89.4%	239.37	120.38	227.85	97.11
Gemma 4 31B	86.9%	259.96	104.17	269.73	107.31
Gemini Robotics ER	77.5%	230.37	91.91	223.15	74.44
Claude Fable 5	71.3%	263.33	146.37	234.89	102.61
Qwen 3.6 27B	37.5%	244.88	128.44	228.61	103.71

Table 4: Behavioral-planning results across all 160 trajectories for GPT-6-Astra ( OpenAI, 2026 ) , Gemma 4 31B ( Google DeepMind, 2026 ) , Gemini Robotics ER ( Gemini Robotics Team, 2025 ) , Claude Fable 5 ( Anthropic, 2026 ) , and Qwen 3.6 27B ( Qwen Team, 2026 ) . All distance metrics are reported in pixels and computed over successfully evaluated outputs.

Figure 5: Left: Predicted tool names and their corresponding bounding boxes generated by GPT 6 Astra, Claude Sonnet 4.6, Gemini 3.1 Pro, and Qwen3.6-27B. Right: Ground-truth task trajectories together with trajectories predicted by Gemini Robotics ER 1.6.

Action	Tool	Kinematic Retarget	BC (Retarget)	ManipTrans (RL)	Dex-X (RL)
Stir coffee	spatula	7.0	1.6	52.2 $\pm$ 2.2	72.7 $\pm$ 0.8
Scoop nuts	ladle	22.2	9.1	62.9 $\pm$ 0.4	66.8 $\pm$ 1.3
Crush banana	mallet	10.2	2.6	53.0 $\pm$ 1.9	55.8 $\pm$ 0.2
Scoop nuts	scooper	31.0	9.8	45.3 $\pm$ 2.0	31.3 $\pm$ 0.6
Crush nuts	pestle	7.0	1.9	16.8 $\pm$ 0.2	46.7 $\pm$ 2.4
Collect sand	squeegee	6.9	2.6	12.6 $\pm$ 0.7	57.5 $\pm$ 1.9

Table 5: Task success rates (%) on the benchmark. Higher is better.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training of 3DGS using 8 cameras and reconstruction of a known camera view

Figure 7: Training of 3DGS using 8 cameras and reconstruction of a novel camera view

	Full-reference			No-reference appeal composite
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	BRISQUE $\downarrow$	NIQE $\downarrow$	PIQE $\downarrow$
SplatFormer	13.09	0.492	0.694	33.11	0.255	0.370	58.38	5.72	44.95
Sparfels	13.72	0.534	0.642	61.16	0.331	0.436	59.39	6.27	67.68
SparseGS	14.64	0.630	0.645	37.61	0.189	0.254	45.85	5.57	70.99
InstantSplat	19.97	0.769	0.428	50.66	0.244	0.403	38.34	4.56	49.93
IS + Difix3D	20.45	0.779	0.364	55.11	0.324	0.456	21.15	4.81	25.13

Appendix

Table 6: 3DGS novel-view synthesis with 8 sparse cameras, benchmarked on three tabletop tool-use scenes ( pestle_crush_nuts , ladle_scoop_nuts , knife_slice_banana ). Each entry is the mean over the three scenes. Methods: SplatFormer ( Chen et al., 2025 ) , Sparfels ( Jena et al., 2025 ) , SparseGS ( Xiong et al., 2023 ) , InstantSplat ( Fan et al., 2024 ) .

Figure 8: Training of 4DGS using 8 cameras and reconstruction of a known camera view

Figure 9: Training of 4DGS using 8 cameras and reconstruction of a novel camera view

	Full-reference (held-out cam 6 )			No-reference appeal composite (novel-view tour)
Pipeline stage	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	BRISQUE $\downarrow$	NIQE $\downarrow$	PIQE $\downarrow$
InstantSplat-4D	19.60	0.715	0.372	51.00	0.22	0.37	39.22	4.87	48.05
IS-4D + Difix3D	19.63	0.728	0.313	62.34	0.32	0.44	30.39	4.84	48.13
IS-4D + Difix3D + GSFix3D	15.97	0.631	0.352	69.92	0.39	0.54	29.55	4.35	53.33

Appendix

Table 7: 4DGS Novel-view synthesis with Sparse 8 cameras benchmarked on three dynamic tabletop tool-use scenes ( mallet_push_toy , spatula_flip_naan , spoon_lift_place_egg ). Full-reference metrics score the held-out cam 6 render of the cam 6 -dropped run against its captured ground truth at 10 sampled timesteps per scene; no-reference metrics score rendered novel-view tour of the 8-cam run.

	Full-reference (8 observed views)			No-reference appeal composite (novel-view tour)
Pipeline stage	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	BRISQUE $\downarrow$	NIQE $\downarrow$	PIQE $\downarrow$
InstantSplat-4D	36.97	0.975	0.044	48.79	0.209	0.354	42.13	5.10	50.61
+ Difix3D	31.87	0.931	0.095	61.93	0.311	0.444	39.25	5.23	55.66
+ Difix3D + GSFix3D	21.11	0.758	0.186	64.69	0.388	0.528	37.78	4.71	49.88

Appendix

Table 8: 4DGS evaluation with all 8 cameras observed, averaged over 20 dynamic tool-use scenes. Full-reference metrics compare each of the 8 training views with its captured ground truth at up to 10 sampled timesteps; no-reference metrics score the rendered novel-view tour.

⬇ System Prompt : You are a robot tool - use reasoning assistant . Select which visible tools could realistically complete the task . Do not default to the most typical tool . User Prompt : Given the scene with multiple laid - out tools , identify which of them could be used to complete the task : "< TASK >". Only select tools that are both visible and plausible for completing the task . Unconventional tools are allowed , but do not include tools that clearly would not work . # only added for Spatial Grounding Tasks Return one bounding box for each tool in the same order as the tools list . Use pixel coordinates [ x1 , y1 , x2 , y2 ] around the visible tool , with x2 and y2 as the bottom - right exclusive coordinates . Return only the canonical base tool name . Do NOT include adjectives , materials , colors , or sizes . Return ONLY the requested JSON object using this schema : { " tools ": ["< exact_tool_name >", "..."],’ # only added for Spatial Grounding Tasks " bboxes ": [[0, 0, 0, 0]] }

Appendix

Table 9: Prompt used for evaluating tool selection and localization capabilities of VLMs.

⬇ System Prompt : You are a robot trajectory planning assistant . Reply with JSON only . User Prompt : Plan a 2 D image - space trajectory for this manipulation task : "< TASK >" Place point 0 on the < TOOL_NAME >. Then return < NUM_POINTS > additional points that describe a plausible trajectory for using the tool to complete the task . Use normalized image coordinates in [ y , x ] format , where each coordinate is an integer in the range 0-1000. Return ONLY a JSON array in the following format : [ {" point ": [ y , x ], " label ": "0"}, {" point ": [ y , x ], " label ": "1"} ] Requirements : - Labels must be strings from "0" through "< NUM_POINTS >". - Include exactly one point for each label . - Do not include code fences or explanatory text .

Appendix

Table 10: Prompt used for behavioral planning and trajectory prediction.

Panel of Figure 10 — Figure 10: Qualitative trajectory generation results. Bounding boxes are color-coded by model: brown for GPT-6-Astra, teal for Claude Sonnet 4.5, purple for Gemini 3.1 Pro, and pink for Qwen3.6-27B.

Panel of Figure 11 — Figure 11: Qualitative trajectory generation results from Gemini Robotics ER across six manipulation scenarios. Red trajectories denote the predicted trajectories generated by Gemini Robotics ER, blue trajectories correspond to the projected tool-center ground-truth trajectories, and green trajectories correspond to the interaction-point transported trajectories. Predicted trajectories generally capture the high-level interaction geometry and intended tool-use behavior, particularly for short-horizon manipulation tasks.

Figure 12: Visualization of the tool retargeting.

Multimodal Dataset

See all 64 papers

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Oct 1, 2026Kyochul Jang, Seohyeon Park, Ohchul Kwon +9Mobile Manipulation Robotic Tool Use
Seoul National University · University of Massachusetts Amherst · Google Research
EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
Sep 30, 2026Shulin Tian, Junsu Kim, Shuai Liu +17Tool-Use Evaluation Tool Use in VLMs
S-Lab, Nanyang Technological University · A*STAR · KAIST +4
RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation
Jun 30, 2026Xinyi Wang, Donghan Li, Zi'Ang Chen +5Robotic Manipulation Bimanual Robotic Manipulation
Fudan University · ByteDance Intelligent Creation · Shanghai AI Laboratory

Oct 1, 2026J/K move · Enter open · S save

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution

Kyochul Jang, Seohyeon Park, Ohchul Kwon, Sangjun Park+8

Seoul National University · University of Massachusetts Amherst · Google Research

Mobile Manipulation Robotic Tool Use Robot Manipulation Benchmarks Humanoid Robotics Embodied Agent Benchmarks

Open paper AskPDF arXiv

As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.

Abstract

Multimodal Dataset

Explore similar work

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation