Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Jusuk Lee; Sungha Kim; Yeonsoo Park; Jonguk Cheon; Yoonkyo Jung; Yongjun You; H. Jin Kim; Jia-Bin Huang

cs.ROOct 8, 2026

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Authors: Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, +3 more

Organizations: Seoul National University · University of Maryland, College Park · All Purpose AI · KAIST

Abstract

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Figures & tables

Figure 1 : From one human video to many dexterous behaviors. Given one human video of a manipulation task, Dex-One2Many learns a dexterous policy entirely in simulation and deploys it zero-shot in the real world. By training across diverse simulated task configurations, the resulting policy executes the task across many unseen configurations beyond the execution shown in the video.

Figure 2 : Overview of Dex-One2Many. (1) From a single human video, a VLM extracts a stage-wise scene graph of object relations. (2) Each stage graph constrains the sampling of object poses and synthesized grasps, yielding verified reset states for every stage. An RL policy trained from these resets with predicate-derived rewards acts in task space through a QP controller. (3) The policy transfers zero-shot to unseen initial poses, goal poses, and grasps on real hardware.

Figure 3 : Using scene-graph predicates for policy learning. (a) Predicates that can be expressed geometrically are mapped to Cartesian-space distance functions. (b) The predicates in a stage-wise scene graph determine which reward terms are assembled into the task reward. (c) The same assembly rule is instantiated for direct and tool-use manipulation from their respective scene graphs.

Figure 4 : Generalization beyond the human video in simulation and the real world. Success rates are reported for configurations shown in the input human video ( Seen , top) and configurations with different initial and goal poses not shown in the video ( Unseen , bottom), in both simulation and the real world. The bottom row illustrates the five evaluated tasks; N/A denotes an experiment unavailable due to a hardware failure.

Figure 5 : Generalization to unseen initial object positions on the Can task. In simulation with the goal fixed, Dex-One2Many succeeds across a broader range of initial positions than the baselines.

Figure 6 : Diverse grasp strategies across dexterous embodiments and object configurations on the Doll task. Across the 33-type grasp taxonomy ( Feix et al., 2015 ) , different embodiments learn different grasp types, and the same embodiment adapts its grasp across object configurations.

	UR3 Wuji 1	UR5e Sharpa	UR5e Wuji 2	UR5e Allegro
Doll	88.4	98.4	97.2	97.2
Can	93.6	97.4	95.2	92.2
Stamp	98.6	96.6	97.0	98.8
Hammer	94.6	94.2	97.6	91.8
Sweep	98.0	96.6	97.0	100.0
Mean	94.6	96.6	96.8	96.0

Table 1 : Simulation performance across diverse dexterous embodiments. Success rates (%) over 500 episodes per task.

Panel of Figure 7 — Figure 7 : Ablation studies. (a) Removing each component. (b) Varying the number of environments.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Model	Input $\rightarrow$ Output	Sec.
Scene-graph extraction	Gemini 3.1 Pro	video $\rightarrow$ entities, parts, stage sequence	B.2
Instance segmentation	SAM 3.1	first frame, visual_phrase $\rightarrow$ object mask	B.3
Mesh reconstruction	SAM 3D Objects	RGB, mask, metric point map $\rightarrow$ metric mesh	B.3
Object pose tracking	FoundationPose++	RGB-D stream, mesh $\rightarrow$ 6D pose	D
Physical parameters	GPT-5	category, size $\rightarrow$ mass, friction	B.4
Part localization	SAM 3	rendered views, part name $\rightarrow$ part region	B.4

Appendix

Table 2 : Pretrained models used in task abstraction, asset preparation, and real-world perception.

Predicate	Reference $A$	Target $B$	Distance metric	Constraint
pre_on_top	top support disk $\mathcal{D}_{A}$	bottom point $p_{B}$	$d_{\mathrm{top}}=d(p_{B},\mathcal{D}_{A})$	$\epsilon_{\mathrm{top}}<d_{\mathrm{top}}\leq\varphi_{\mathrm{top}}$
on_top				$d_{\mathrm{top}}\leq\epsilon_{\mathrm{top}}$
pre_inside	interior box $\mathcal{B}_{A}$	root point $p_{B}$	$d_{\mathrm{in}}=d(p_{B},\mathcal{B}_{A})$	$\epsilon_{\mathrm{in}}<d_{\mathrm{in}}\leq\varphi_{\mathrm{in}}$
inside				$d_{\mathrm{in}}\leq\epsilon_{\mathrm{in}}$
pre_contact	center $p_{A}$	working point $p_{B}^{\mathrm{work}}$	$d_{\mathrm{con}}=\\|p_{A}-p_{B}^{\mathrm{work}}\\|_{2}$	$\epsilon_{\mathrm{con}}<d_{\mathrm{con}}\leq\varphi_{\mathrm{con}}$
contact				$d_{\mathrm{con}}\leq\epsilon_{\mathrm{con}}$

Appendix

Table 3 : Predicate vocabulary. Geometric features, distance metrics, and constraints used to instantiate the predicates in our tasks. Paired pre-relation and terminal predicates share the same distance metric and differ only in their constraints, except for pre_grasp / grasp .

Figure 8 : Single human demonstrations for the five tasks. Each row shows the human video used to specify one task, with frames ordered from left to right over time.

Task	Description	Objects (role)	$K$	Predicate chain (stage $1\rightarrow K$ )	Type
Doll	Pick up a Pikachu plush toy and place it inside an orange pot.	pikachu (held), pot † (site)	5	$\varnothing$ $\rightarrow$ pre-grasp (hand, pikachu) $\rightarrow$ grasp (hand, pikachu) $\rightarrow$ pre-inside (pikachu, pot) $\rightarrow$ success: inside (pikachu, pot)	direct
Can	Pick up the coke can and place it in a bucket.	can (held), bucket † (site)	5	$\varnothing$ $\rightarrow$ pre-grasp (hand, can) $\rightarrow$ grasp (hand, can) $\rightarrow$ pre-inside (can, bucket) $\rightarrow$ success: inside (can, bucket)	direct
Stamp	Grasp the stamp and press it down onto the stack of papers.	stamp † (held), paper (site)	5	$\varnothing$ $\rightarrow$ pre-grasp (hand, stamp) $\rightarrow$ grasp (hand, stamp) $\rightarrow$ pre-on_top (stamp, paper) $\rightarrow$ success: on_top (stamp, paper)	direct
Hammer	Use a hammer to strike a peg inserted into a block.	hammer † (held), peg † (acted-on), block † (site), holder † (fixture)	6	fixed_to (hammer, holder) $\rightarrow$ pre-grasp (hand, hammer) $\rightarrow$ grasp (hand, hammer) $\rightarrow$ pre-contact (hammer, peg) $\rightarrow$ contact (hammer, peg) $\rightarrow$ success: seated (peg, block)	tool-use
Sweep	Sweep the toy cat into the dustpan using the brush.	brush (held), toy (acted-on), dustpan (site)	7	$\varnothing$ $\rightarrow$ pre-grasp (hand, brush) $\rightarrow$ grasp (hand, brush) $\rightarrow$ pre-contact (brush, toy) $\rightarrow$ contact (brush, toy) $\rightarrow$ pre-inside (toy, dustpan) $\rightarrow$ success: inside (toy, dustpan)	tool-use

Appendix

Table 4 : Task suite. Object roles: held is grasped by the robot, site receives another object, acted-on is manipulated only through a tool, and fixture holds an object at the start. The predicate chain extracted by the VLM defines

K

stages, each with its own reset distribution. Objects marked with † use a CAD mesh instead of a reconstructed one (Appendix B.3 ).

Field	Content	Used for
task
name , description	Task name and one-line summary	Generated first to condition the subsequent fields; not used by the pipeline
objects[]
id	Identifier of each task-relevant object	Nodes; key for all per-object data
category	Object class	Hint in the physical-parameter query
visual_phrase	Text description of the object’s appearance	Object segmentation

Appendix

Table 5 : Structure of the VLM output and the use of each field in our pipeline. Indented entries are fields of the object above them, and [] marks a list with one entry per object, mount, or stage.

Task	Video [s]	Latency [s]	Cost [$]
Doll	7.07	21.2	0.045
Can	5.30	11.5	0.024
Stamp	4.97	12.0	0.026
Hammer	4.00	18.7	0.039
Sweep	5.77	18.5	0.037
Total	27.11	81.9	0.171

Appendix

Table 6 : VLM query cost per task.

Step	Tool	Parameters
Convex decomposition	CoACD ( Wei et al., 2022 )	concavity threshold 0.05; resolution 2000; MCTS 20 nodes, 150 iterations, depth 3; $rv_{k}=0.3$ ; manifold preprocessing only for non-manifold input (resolution 50); hull merging on; no hull-count limit; minimum part volume $10^{-10}$ ( -mv ); random seed from the system clock
Watertight remesh	CoACD manifold mode (OpenVDB ( Museth et al., 2013 ) )	applied to the normalised mesh; dual-marching-cubes level set 0.1
Simplification	ACVD ( Valette and Chassery, 2004 )	2000 vertices, gradation 1.5, manifold output

Appendix

Table 7 : Mesh post-processing.

Step	Setting
Views	34 views: elevations $-35^{\circ},10^{\circ},35^{\circ},60^{\circ}$ $\times$ 8 azimuths ( $45^{\circ}$ steps), plus top ( $85^{\circ}$ ) and bottom ( $-85^{\circ}$ ); rendered twice (textured and uniform grey); $768{\times}768$ px, vertical field of view $45^{\circ}$ , camera distance $2.6\times$ the half bounding-box diagonal
Detection	SAM 3; the part string is one text concept; instance score floor 0.15, mask threshold 0.5
Negative concepts	grasping only: blade, cutting edge, spike, bristles ; their masks are dilated by 12 px and removed
Fusion	a vertex is visible in a view if its depth is within $2\%$ of the bounding-box diagonal of the z-buffer; score $=$ positive votes / visible views, over views with at least one detection; vertex kept if seen in $\geq 2$ views and score $>0.3$
Transfer to mesh	each simulation-mesh face takes the maximum score of the nearest triangle of the reconstructed mesh; face kept if score $\geq 0.25$ ; grasping region: largest connected component
Outputs	per-face mask and object-frame bounding box (centre, extents) of the grasping region; for grasp synthesis on the table, the mask is restricted to faces whose lowest vertex is $\geq 5$ mm above the table

Appendix

Table 8 : Localization of grasping and working regions.

Figure 9 : Scene-graph-grounded observation. The policy observation is constructed from scene-graph nodes and their relative 6D-pose edges for (a) direct and (b) tool-use manipulation.

Figure 10 : Scene-graph-constrained reset generation. Given the stage-wise scene graph and the assets, we instantiate each task stage as diverse reset states. The object and hand assets provide geometry and physical properties, while functional part nodes identify regions such as grasp and working regions. Together, these are used to construct a grasp set, restricted to the specified grasp region when available. For each stage, predicates in the scene-graph act as generative constraints: geometric predicates determine where related entities can be sampled, while the grasp predicate samples a valid hand–object configuration from the grasp set. Repeating this process produces diverse simulation reset states that satisfy the relations specified by each stage graph.

Figure 11 : Physical verification of reset candidates in the Sweep task. (1) and (2) (top: pass, bottom: fail), and (3) is enforced over grasp types. (1) Scene collision : the robot makes no invalid contact with the table or other objects; the failing candidate collides with the dustpan. (2) Reset stability : objects settle under gravity and the grasped object stays in the hand under external wrenches; the failing grasp drops the brush. (3) Goal compatibility : the goal relation remains attainable with the candidate grasp; the failing candidate cannot bring the toy cat inside the allowed region of the dustpan.

Figure 12 : QP constraints address a hardware-specific sim-to-real gap. Hand–table contact is benign in simulation because coupler disengagement is not modeled, but the same contact can disengage the Wuji 1 hand coupler on the physical robot. Without the QP low-level controller (top), the learned policy encounters this failure mode. With the QP controller (bottom), a hand–table clearance constraint prevents the collision without retraining the task-space policy.

PPO		Network
Parallel environments	$49{,}152$	Actor recurrence	LSTM, 1 layer, width $1024$
Rollout horizon	$32$ steps ( $3.2$ s)	Actor recurrent norm	Layer norm
Learning epochs	$5$	Actor MLP	$[1024,1024,512,512]$
Mini-batches	$4$	Critic	MLP $[1024,1024,512,512]$
Clip range	$0.2$	Activation	ELU
Discount $\gamma$	$0.99$	Action distribution	Gaussian, learned $\log\sigma$

Appendix

Table 9 : Policy optimization configuration. We use the same configuration for all tasks and embodiments. The learning rate follows an adaptive KL schedule initialized at

10^{-4}

and clipped to

[10^{-5},10^{-2}]

Property	How it is varied
Physical properties
Object mass	Between the minimum and maximum mass estimated for each object
Object static friction	Up to 10% above or below the estimated friction
Object dynamic friction	Up to 10% above or below the estimated friction
Table friction	Static: 0.3–0.6; dynamic: 0.2–0.5
Table mass	0.5–1.5 times the original mass

Appendix

Table 10 : Domain randomization. Multipliers are relative to the original simulation values. Ranges are sampled uniformly unless otherwise stated.

Figure 13 : Generalization to unseen initial object positions on the Doll task. With the goal fixed, Dex-One2Many succeeds over a substantially broader range of initial positions than the baselines.

Figure 14 : Generalization to unseen initial object positions on the Can task. With the goal fixed, Dex-One2Many succeeds over a substantially broader range of initial positions than the baselines.

Figure 15 : Generalization to unseen initial object positions on the Stamp task. With the goal fixed, Dex-One2Many succeeds over a substantially broader range of initial positions than the baselines.

Figure 16 : Generalization to unseen initial object positions on the Hammer task. With the goal fixed, Dex-One2Many succeeds over a substantially broader range of initial positions than the baselines.

Figure 17 : Generalization to unseen initial object positions on the Sweep task. With the goal fixed, Dex-One2Many succeeds over a substantially broader range of initial positions than the baselines.

Figure 18 : Diverse grasp strategies for the Doll task. Learned grasp types vary across dexterous embodiments and object configurations.

Figure 19 : Diverse grasp strategies for the Can task. Learned grasp types vary across dexterous embodiments and object configurations.

Figure 20 : Diverse grasp strategies for the Stamp task. Learned grasp types vary across dexterous embodiments and object configurations.

Figure 21 : Successful simulation rollouts across diverse dexterous embodiments for the Doll task. For each of four dexterous embodiments, we show three successful rollouts from distinct task configurations. Each row corresponds to one configuration, and columns show the execution over time from left to right.

Figure 22 : Successful simulation rollouts across diverse dexterous embodiments for the Can task. For each of four dexterous embodiments, we show three successful rollouts from distinct task configurations. Each row corresponds to one configuration, and columns show the execution over time from left to right.

Figure 23 : Successful simulation rollouts across diverse dexterous embodiments for the Stamp task. For each of four dexterous embodiments, we show three successful rollouts from distinct task configurations. Each row corresponds to one configuration, and columns show the execution over time from left to right.

Figure 24 : Successful simulation rollouts across diverse dexterous embodiments for the Hammer task. For each of four dexterous embodiments, we show three successful rollouts from distinct task configurations. Each row corresponds to one configuration, and columns show the execution over time from left to right.

Figure 25 : Successful simulation rollouts across diverse dexterous embodiments for the Sweep task. For each of four dexterous embodiments, we show three successful rollouts from distinct task configurations. Each row corresponds to one configuration, and columns show the execution over time from left to right.

Explore similar work

Sep 7, 2026cs.RO

Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction

Human videos are an abundant source of dexterous manipulation behaviors, but they lack tactile information that is crucial for contact-rich interaction. This raises a fundamental question: can robots learn deployable visual-tactile dexterous manipulation policies from human video demonstrations without robot-side data collection? We present DEX-X, a framework for learning visual-tactile dexterous manipulation from human videos through simulation. Our key insight is that simulation can serve as a tactile completion engine. Given monocular human demonstrations, DEX-X reconstructs hand-object interactions in simulation, where physically grounded contact dynamics provide tactile supervision unavailable in the original videos. Leveraging this recovered tactile information, we train visual-tactile dexterous manipulation policies and distill them into deployable policies operating on point-cloud observations and tactile sensing. We demonstrate zero-shot sim-to-real transfer on a dexterous hand-arm platform across diverse grasping and contact-rich tool-use tasks. The teacher policy achieves 65.9% average success across six task categories in simulation, while the distilled visual-tactile policy achieves 93% success on real-world cube picking and 53% on the challenging table-cleaning task. Zero-shot generalization to unseen object geometries is also observed on object-picking tasks. Our results suggest that simulated interaction is a key bridge between human videos and deployable dexterous manipulation policies, providing the missing physical supervision needed for scalable robot skill learning from Internet-scale human video data.

Ruoqu Chen, Feixiang Ruan, Liu Cao +9

Tsinghua University · Shanghai Qizhi Institute · Sharpa +2

Sim-to-Real Transfer Robotic Manipulation Robot Imitation Learning Robot Learning

Ask PDF

Jun 9, 2026cs.RO

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.

Beomjun Kim, Seong Hyeon Park, Seunghoon Sim +3

KAIST

Dexterous Manipulation Cross-Embodiment Robot Learning Robotic Manipulation Robot Imitation Learning

Ask PDF

Jun 15, 2026cs.RO

V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos

Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from monocular videos represents a promising frontier in embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial alignment and physical consistency. Evaluations on the TACO and OakInk benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.

Kaihan Chen, Yanming Shao, Haifeng Ji +2

Zhejiang University, Hangzhou, China · Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Laboratory, Shanghai, China

Cross-Embodiment Robot Learning Robotic Manipulation Learning from Demonstration Robot Imitation Learning

Ask PDF