cs.ROOct 13, 2026

FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

Authors: Zhi Cao, Yining Hong, Jiaolong Yang, Yejin Choi, Li Fei-Fei, Jiajun Wu, Huang Huang

Organizations: University of Michigan · Stanford University · Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements. Dexterous manipulation requires not only visual and proprioceptive understanding but also the ability to anticipate and regulate interaction forces. To scale force-aware human-robot co-training, we introduce EgoForce, a large-scale human tactile dataset capturing force-rich manipulation experience, and combine it with robot demonstrations for pretraining. We present FACT, a robot foundation model that cascades from prospective visual planning to force-aware refinement for dexterous manipulation. FACT comprises two jointly trained components. A Track-Guided Coarse Planner jointly predicts actions, future video, and visual tracks. Future video and visual tracks also provide a shared task representation that facilitates cross-embodiment knowledge transfer and coarse action generation. Building on this coarse plan, a Force-Action Joint Refiner jointly predicts future force and refines the coarse action, allowing anticipated force and action to shape each other during refinement. We evaluate the model on ten tasks requiring fine-grained coordination and force feedback, including tool use, musical instruments, and precision laboratory procedures. Using as few as 10 robot trajectories with 90 human trajectories for fine-tuning, FACT outperforms both baselines, with success-rate gains of up to 60 percentage points over T-Rex. Ablations show that force supervision in human pretraining data is crucial for downstream robot performance after fine-tuning, while visual track prediction and training with future-force prediction improve dexterous execution. Experiments varying the human-robot data ratio during fine-tuning further suggest that FACT learns from human demonstrations nearly as effectively as from robot demonstrations.

Figures & tables

Fig. 2: EgoForce task and skill distribution. Left: episode share by task category, with over two thirds of the corpus covering contact-rich fine manipulation across diverse task categories. Right: atomic-skill annotations on a log scale, showing a broad distribution of dexterous, force-dependent skills beyond simple transport.
Fig. 3: FACT architecture. The Track-Guided Coarse Planner jointly denoises future video, visual tracks, and a coarse action chunk with a block-causal video DiT; per-token noise conditioning lets clean context and noisy targets coexist. At the τ=0.4\tau{=}0.4 handoff, video and tracks are read out and the coarse hidden states are frozen. The Force–Action Joint Refiner reads them through cached cross-attention and co-denoises the remaining action ( τa ⁣:0.4→0\tau_{a}\!:0.4{\to}0 ) with continuous future fingertip force ( τf ⁣:1→0\tau_{f}\!:1{\to}0 ), using observed tactile signals as clean context.
Fig. 4: Evaluation randomization. Overlaid initial frames from six evaluation tasks illustrate rollout-level variation in object positions and poses.
    FACT (ours)
#Taskπ0.5\pi_{0.5}T-Rexw/o force predw/o track predfull
Binary tasks: success rate (%) ↑\uparrow
1Lab pipetting00302040
2Open test-tube cap1020408080
3Open microcentrifuge flip cap020204050
4Tie shoelaces0002020
TABLE I: Real-robot evaluation on the ten-task suite (scoring in Section V-A ). For pipette volume, the indented second row gives the mean absolute error in μ\mu L over the engaged trials. w/o force pred and w/o track pred remove future-force prediction and visual track prediction during training, respectively.
Fig. 5: Qualitative examples. FACT executes contact-rich behaviors requiring dexterous multi-finger coordination, precision fingertip micro-manipulation, and in-hand multi-object grasping.
#Taskhuman w/ force+robot (full)human w/o force+robotrobot only
Binary tasks: success rate (%) ↑\uparrow
1Lab pipetting40200
2Open test-tube cap80020
3Open microcentrifuge flip cap50100
4Tie shoelaces2000
5Shuffle cards30100
TABLE II: Pretraining recipe ablation on all ten tasks.
Fig. 6: Robot-to-human mixture in fine-tuning. Performance versus the robot-to-human ratio of the 100-trajectory per-task fine-tuning mixture (10:90, 30:70, 50:50).
  robot : human data
#Task10 : 010 : 90
Binary tasks: success rate (%) ↑\uparrow
1Tie shoelaces1020
2Shuffle cards1030
Volume adjustment: engaged trials of 10 ↑\uparrow
3Pipette volume45
TABLE III: Contribution of human fine-tuning data on three dexterous tasks: ten robot trajectories alone (10:0) versus the 10:90 robot-to-human mixture, scored as in Table I .
Fig. 7: Scaling with human data. Validation loss (MSE) versus hours of human demonstration data (100, 500, 1k, 1.5k, and 2k hours on a linear axis), mixed with robot teleoperation data, with and without tactile supervision. Tactile supervision lowers the loss at every scale and the gap persists as data grows.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Track-Guided Coarse Planner (Flow Matching)
BackboneWan2.2-TI2V-5B video DiT
Hidden dimension3072
Number of layers30
Max sequence length2,518 tokens
Action dimension62
Action chunk16
Appendix
TABLE IV: Model and training configuration of FACT .
Fig. 8: EgoForce collection system and data. Left : collection hardware (egocentric stereo RGB camera, wrist RGB cameras, and piezoresistive tactile gloves) and front and side views of an operator wearing the system. Middle : the FusionX multimodal collection system (TouchTronix Robotics, NY), illustrated with synchronized head and wrist views and tactile readings from a shirt-folding recording; the head view includes estimated hand poses and image-space hand trajectories, which are not motion-capture ground truth. Right : twelve examples from the collection environments, each showing the egocentric view and the tactile maps of the left and right hand (finger labels T, I, M, R, L for thumb, index, middle, ring, and little finger). Heatmaps show per-taxel forces in newtons on a shared 0–8 N scale with square-root color mapping.
Fig. 9: Real-world setup and data-collection interfaces. Top: the bimanual station, with two I2RT YAM Ultra arms driven by a YAM Box controller, a Sharpa Wave 1 hand on each arm, a RealSense D435 head camera, and a ZED Mini camera on each wrist. Bottom left: the teleoperation interface, Manus gloves with a VIVE Tracker 3.0 on each wrist. Bottom right: the human demonstration interface used for the fine-tuning data, Wuji tactile gloves with a RealSense D435 wrist camera and a VIVE Tracker 3.0 on each wrist.
Fig. 10: Full input modalities of FACT on Task 10: Adjust Pipette Volume. Text Instruction: “A pipette set to 420 μ\mu L is on the table. Hold the pipette body with your left hand and turn the volume dial with your right thumb and index finger until the display reads 400 μ\mu L.” Shown at one instant of an autonomous rollout while the dial is being turned: the head-camera view, the left and right ZED Mini wrist views, and the ten fingertip wrenches (rows R and L, one panel per finger; the dot gives the tangential force and the color the normal force), where the right thumb and index contacts on the dial appear while the left hand holds the pipette body. Together with the robot state, these are all the inputs the policy receives.
Fig. 11: Task 1: Lab Pipetting.
Fig. 12: Task 2: Open Test-Tube Cap.
Fig. 13: Task 3: Open Microcentrifuge Flip Cap.
Fig. 14: Task 4: Tie Shoelaces.
Fig. 15: Task 5: Shuffle Cards.
Fig. 16: Task 6: Multi-Object Grasp.
Fig. 17: Task 7: Use Screwdriver.
Fig. 18: Task 8: Play Harp.
Fig. 19: Task 9: Play Xylophone.
Fig. 20: Task 10: Adjust Pipette Volume.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Explore similar work

CardsList
Aug 16, 2026

ReForce: Learning Force-aware Retargeting for Dexterous Manipulation

Human demonstrations offer a scalable data source for dexterous manipulation, but transferring them to robot actions remains challenging due to the embodiment gap. Today's retargeting is mostly kinematic, yet manipulation is decided by force, which governs how the hand interacts with the object and how the object moves. In this paper, we present ReForce, a Force-aware Retargeting method that turns human motion and forces into robot actions that reproduce the intended contact. ReForce predicts a residual on the kinematically retargeted action to reach the desired force, using a general force tracker trained on large-scale simulation interactions. It supports both online force-aware teleoperation and offline data translation. In simulation and on real hardware, ReForce achieves lower force-tracking error and stronger multi-finger contact engagement on contact-rich tasks such as paper-cup grasping and tongs manipulation.

Yuhang Wu, Lingqi Zeng, Changwei Jing +2

UC San Diego

Motion RetargetingRobotic ManipulationForce-Controlled Robotic ManipulationContact-Rich Robotic Manipulation
AskPDF
Feb 18, 2026

EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While prior work demonstrates human to robot transfer in constrained settings, it is unclear whether large scale human data can support fine grained, high degree of freedom dexterous manipulation. We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data. We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss. This validation loss strongly correlates with downstream real robot performance, establishing large scale human data as a predictable supervision source. Beyond scale, we introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training. This enables strong long horizon dexterous manipulation and one shot task adaptation with minimal robot supervision. Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand, and transfers effectively to robots with lower DoF hands, indicating that large scale human motion provides a reusable, embodiment agnostic motor prior.

Ruijie Zheng, Dantong Niu, Yuqi Xie +12

NVIDIA · University of California, Berkeley · University of Maryland

Robotic ManipulationRobot LearningTransfer LearningLong-Horizon Robotic ManipulationCross-Embodiment Robot LearningData-Efficient Robot Learning
AskPDF
Nov 21, 2025

METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model

Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.

Yankai Fu, Ning Chen, Junkai Zhao +5

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence

Robotic ManipulationVLMs for RoboticsRobot LearningGeneralization in Robotic ManipulationCross-Embodiment Robot LearningRobustness of VLA Models
AskPDF