In dexterous in-hand manipulation, learning-based methods achieve strong task performance with increasing generalization to unseen objects. A complementary advantage of model-based methods, central to this work, is online adaptation to new task specifications without policy retraining, given suitable system models. However, due to the complexity of physical contacts and limited sensing modalities, existing model-based methods still struggle with modeling errors, even when using advanced contact models or optimization schemes, which limit their real-world applications. To advance the effectiveness and robustness of model-based contact-rich in-hand manipulation, this paper proposes a novel bi-level framework with tactile incorporation to mitigate these limitations. Specifically, at the high level, reference finger motions and contacts are jointly generated via contact-implicit model predictive control. The high-level module enables online reactive planning and disturbance recovery. At the low level, these references are concurrently tracked through a hand force-motion model and tactile feedback. The low-level module compensates for modeling errors and enhances manipulation robustness. The complementarity between the two levels is manifested in two dimensions: 1) subsystem decomposition, with a slow optimization-based planning layer and a fast tactile-reactive control layer; and 2) functional decomposition, combining task-level optimality with contact-level stability. Extensive experiments demonstrate that our approach outperforms existing model-based methods in accuracy and robustness, while supporting online replanning during execution. Our method successfully completes a diverse set of challenging tasks in real-world environments, even under appreciable external disturbances and modeling uncertainties.
Figures & tables
Figure 1: The proposed tactile-informed bi-level framework advances model-based in-hand manipulation under real-world challenges, including external disturbances, perception noise, and modeling uncertainties. With a bi-level architecture, it explicitly resolves the incompatibilities between tactile incorporation and optimization-based control. The framework couples real-time motion–contact planning with tactile-informed local corrections, enabling reactive, human-like finger gaiting, while mitigating modeling errors and ensuring contact-level stability. This combination results in robust, precise, and multi-task in-hand manipulation.
High-Level Module
x
state of the full hand-object system [xu;xa]
xu
object pose (the orientation is parameterized by quaternion)
xa
hand joint positions
x+
state of the next time step
xinit
initial state of each optimization iteration
u
control input (i.e., delta desired joint positions)
Table 1: Notations Used in this Paper
Figure 2: Proposed bi-level framework for robust in-hand manipulation with online motion-contact replanning toward real-world challenges. (A) The user inputs the desired object motion, hand grasp pose (i.e., nominal configuration), and corresponding models. (B) The high-level motion-contact planner employs contact-implicit MPC to generate motion-contact references from the initial state x0 , state reference xref , and the previous iteration’s solution X∗,U∗ . (C) The low-level tactile-reactive tracking controller leverages adaptive-weighted MPC-based HFMC to synergistically track motion–contact references with minimal mutual interference. (D) Both levels complement each other, forming a systematic, tactile-inclusive, multi-task framework for in-hand manipulation.
Figure 3: Illustration of the inter-level coordination. The top figures show the high-level motion-contact planner, which generates finger-gaiting patterns online with embedded contact information (visualizing only the index finger and contact forces). The right-side figures show results without (w/o) and with (w/) motion correction enabled by the proposed low-level controller. The top-right figure illustrates how modeling errors lead to the force-at-a-distance effect ( Anitescu, 2006 ) , where non-zero planned forces appear even when contact is inactive. Modeling errors can be mitigated through low-level motion-contact tracking with tactile incorporation (shown in the bottom-right figure). The bottom-left figure shows how motion correction is achieved by deforming the command fingertip trajectory (traj.). Best viewed in color.
Figure 4: Illustration of trajectory shift for action smoothness. The high-level MPC computes optimal state sequence X∗ with delays, shown by light green curves. The low-level module interpolates the previous solution Xold∗ until a new one is received at trecv . The interpolated trajectory is shown in dark green, and the arrow indicates discontinuity. To reduce the discontinuity and ensure smoother hardware commands, we shift the new solution to Xnew∗ . Note that due to multiple delays, the real system states xreal always lag the commanded states.
Figure 5: Notations of the force-motion model. The finger uses joint proportional-derivative (PD) control with stiffness KP and damping KD . Due to controller compliance, the desired fingertip location differs from the actual location. At the contact point, the environment stiffness Ke and the finger’s Cartesian stiffness Kr result in the equivalent stiffness Kˉs . The single contact stiffness relates the desired motions dpc,d with the contact force λext . The forces and motions of all contacts are correlated due to the resultant displacement δxu of the object’s center of mass (COM). The force controlled direction n^ and motion controlled directions {t^1,t^2} are determined using the contact normal. This figure has been adapted from the conference version of this work ( Jiang et al., 2024 ) .
Figure 6: Comparison of global (e.g., the CQDC model) and local force–motion models. The global model, assuming nominal object geometry (geom.), defines forces based on absolute finger positions, whereas the local model conditions forces on relative finger displacements. Consequently, only the local model forms a tactile feedback pathway, iteratively updating finger positions ( Δx ) until contact is achieved. In contrast, the global model computes fixed finger positions, which may not result in actual contact.
Figure 7: Snapshots of the environment-constrained manipulation tasks used for evaluation in simulation and on hardware. We evaluate our method on two dexterous hands—the Allegro Hand and the LEAP Hand—across a variety of tasks to demonstrate the task-level adaptability of the proposed model-based framework to different manipulation setups.
Figure 8: Illustration of finger gaiting patterns across different methods. The colored patches denote time intervals during which the contact distance of each finger falls below 1 mm. Representative snapshots are shown on the right. For Jin (2025) and Pang et al., the fingers move almost simultaneously with periodic manual regrasping, as reflected by the gaps, whereas the proposed framework exhibits natural, human-like finger gaiting behavior. For a more intuitive visualization of this phenomenon, we refer interested readers to the attached video .
Figure 9: Task error distribution for the Rotate Sphere task with noisy object orientation. The x-axis labels indicate the standard deviation of the added Gaussian noise. The bars represent mean and extreme values, whereas the gray stripes show the intervals between the first and third quartiles.
Figure 10: Distribution of the task error across 100 trials in the Rotate Sphere task, under modeling errors. We use a consistent radius of r=6.0cm in the planning model across different groups. The x-axis labels indicate the actual radius used in the MuJoCo simulation ( 5.0∼6.4cm ). The control group (CG) has zero modeling error. Groups L1 to L2 have a larger radius, whereas groups S1 to S5 have a smaller radius for the simulated sphere.
Figure 11: Performance comparison between the proposed adaptive weighting scheme and the fixed-weight baseline on the Rotate Sphere task under modeling errors. The modeling error is introduced by using different sphere radii in simulation and the planning model, as in Fig. 10 . Markers denote mean values, and error bars represent standard deviations.
Figure 12: Distribution of the relative force tracking error when executing 100 grasps generated using BODex ( Chen et al., 2025b ) . The scatter points represent outliers that lie beyond 1.5 times the inter-quartile range (IQR) from the box. Points that exactly fall on the 100% line correspond to zero contact force (i.e., missing contacts), while points above the 100% line indicate excessively large contact forces (i.e., grasp failure). Examples of the hand configuration at the pre-grasping stage are shown on the right.
Figure 13: Results of the in-grasp object movement experiment. Top: Task snapshot and waypoints . The marker at the top center of the cylinder successively moves to the eight corners of the cube and back to the initial position. Middle: Normal contact force . Dashed lines represent the desired values, whereas solid lines show the execution results. The three dark lines indicate the results of the proposed controller, and the three light lines represent the baseline results. Bottom: Position tracking error . The local minima indicate tracking errors when the marker reaches the waypoints.
Figure 14: Experimental setup: (a) The LEAP Hand is mounted on the flange of a UR5 arm. An AprilTag and a RealSense D405 camera are used for perception. (b) The fingertips of the LEAP Hand are replaced with four Tac3D tactile sensors.
Figure 15: Experimental results for the Open Door task with human disturbance. (a) Snapshots from two views. (b) Door handle rotation in different trials. (c) Planned and real contacts (Trial 4) represented as time intervals during which the contact forces exceed a given threshold. (d)(e) Planned and real contact forces.
Figure 16: Real-world Open Door task—Comparison with the ablation without contact tracking. Here, ‘ours’ refers to Ours (2 nd ) , and ‘openloop’ refers to Ours (2 nd , openloop) in Tab. 4 . The curves represent the mean handle rotation, while the shaded areas indicate the standard deviation. The dashed horizontal line marks the target rotation.
Figure 17: Snapshots of the Open Door task under size and shape uncertainties. The proposed framework with contact tracking establishes the planned contacts despite substantial modeling errors, while the ablation without contact tracking shows insufficient contact force or even missed contacts.
Figure 18: (a) Real-world door handles of different sizes and shapes used in the Open Door task. Handles A ( ϕ=6 cm) and B ( ϕ=8 cm) introduce object-size uncertainties, while handle C—an irregular shape with a maximum diameter of 10 cm—represents partially defective objects. (b) Handle rotation angles over time for different handles and control methods, where ‘w/ contact tracking’ and ‘w/o contact tracking’ refer to Ours (2 nd ) and Ours (2 nd , openloop) in Tab. 4 , respectively.
Figure 19: Experimental results for the Rotate Card task with disturbance. (a) Snapshots of the task. The white border represents the target pose. Rotational and translational disturbances are exerted by the human operator. (b)(c) Object rotation (around the x-axis) and translation (on the yOz plane) refer to the coordinate system shown in the first snapshot in (a). (d) Planned and measured contact forces of the ring, middle, and index fingers (from the top down).
Figure 20: Experimental results for the Slide Board task with disturbance. (a) Snapshots of the task. The white border represents the target pose. External disturbances are exerted by the human operator. (b)(c) Object rotation and translation refer to the coordinate system shown in the first snapshot in (a), ’z’ refers to translation along the z-axis, and ’rx’ refers to the rotation around x-axis. (d) Planned and measured contact forces of the ring, middle, and index fingers (from the top down).
Figure 21: Experimental results for the Open Box task. At the top are snapshots of the simulation and the real-world experiments on four boxes A ∼ D. The first row shows the initial state, and the second row shows the state when the box is opened. The yellow and cyan boxes represent the initial and opened positions of the lid, respectively. (a) ∼ (d) Lid rotation of different boxes.
Figure 22: Relative force tracking error and snapshots of the grasping experiment. The objects are (a) banana, (b) mango, (c) onion, (d) pear, and (e) pepper. Scatter points denote outliers beyond the 1.5-IQR whiskers.
Figure 23: Results of the in-grasp object movement experiment. The object tracks target positions or poses through rolling contacts while maintaining contact-force regulation. Left : relative force tracking errors of the three contacting fingers. Middle : snapshots of target pose tracking. Right : object tracking errors for the position-only (3D Pos) and full-pose target (6D Pose) groups.
Figure 24: Results of the prehensile in-hand manipulation task with finger gaiting. (a) Object yaw angle over time for all six trials; for Trial 0, the object’s x-y position is also shown. Trial 5 is conducted with the ablation Ours (2 nd , openloop) in Tab. 4 without contact tracking. (b) Snapshots and finger contact forces for Trial 0. Solid lines denote desired contact forces, and dashed lines denote measured contact forces.
Figure 25: Top-view snapshots of the same in-hand manipulation task as in Fig. 24 , with the object replaced by a cube. The detected marker frame is visualized.
Figure 26: Elapsed time of high-level motion-contact planning during key sub-steps. Here, calc and calc_diff represent the forward dynamics and gradient computation of the CQDC model, respectively. The total height of the bar indicates the average iteration time for each task.
Figure 27: Rotation speed and solution time per iteration across MPC horizons in the Rotate Valve task. The error bars indicate standard deviations. The dashed lines represent average values.
Figure 28: Effects of κ on optimization performance, including the cost value, the door handle’s rotation (yaw angle), and the action norm. Warmer colors indicate a weaker smoothing effect and less nonphysical behavior.
Figure 29: Additional results of the simulated Rotate Sphere task regarding complex geometries. The sphere is replaced with 10 complex objects from the DGN’s ( Wang et al., 2023 ) dataset and rotated in place toward 20 target orientations. Task snapshots, the object geometries with decomposed convex parts, their associated success rate and error distributions are visualized. Most trials achieve precise manipulation with orientation errors below an 8° threshold.
Figure 30: Parameter sweep results for the fingertip orientation tracking weight wpori , with the best-performing wp and wΛ in Fig. 31 fixed. All three metrics show no significant change as wpori varies, and a moderate value of 1.8 is selected.
Figure 31: Heatmap of the parameter sweep on the Rotate Sphere task, where wp and wΛ denote the scalar weights for fingertip position tracking and contact force tracking, respectively, in the fixed-weight baseline. The selected parameter combination, wp=30 and wΛ=6 , which yields the lowest task error, the highest success rate, and moderate joint acceleration, is marked in red.
Figure 32: Key variables of the adaptively weighted low-level controller during three representative tasks. From top to bottom: desired contact force magnitude ∥λextref∥ (dashed line: threshold λ ; blue shading: ∥λextref∥≥λ ); ∥W^p∥F and ∥W^Λ∥F , Frobenius norm of the adaptive weights respectively for tracking fingertip pose and contact force; ∠(n^,n^0) , angle in degrees between the current contact normal and its initial value at t=0 ; n^⊤Δp , signed position tracking error along the contact normal; ∥T^T^⊤Δp∥ , tangential position tracking error; and contact distance ϕ (dashed curve: contact distance when merely following the high-level reference). Here, Δp is the translational component of FK(q)⊖FK(qref) .
Figure 33: Additional results of the open door task with different relative wrist-door poses. (a) The door is rotated 30 degrees counterclockwise (case 1). The ring finger is not used. (b) The door is rotated 30 degrees clockwise (case 2). The index finger is not used. (c) Door handle rotation in different trials.
Figure 34: (a) Snapshots of the Rotate Valve task. The red circles mark the fingers in contact. The fingers detect contacts within their reachable space, sequentially touching the valve. Due to the limited sensing regions of the Tac3D tactile sensors, we replace them with spherical fingertips and assume zero contact forces at all fingertips to enable the proposed method. However, this setup does not fully demonstrate the low-level module’s effectiveness in contact tracking. (b) Valve rotation angle relative to time.
Fusing tactile signals has proven effective for contact-rich manipulation, enabling robots to perceive contact states and adapt to rapidly changing physical interactions. Yet effectively integrating tactile feedback into dexterous manipulation remains underexplored. In this work, we introduce ReTouch, a vision-language-action model (VLA) that supports contact-rich dexterous manipulation through tactile predictions continually refined online using execution-time feedback. ReTouch builds on two main innovations for tactile representation and closed-loop action generation. First, its Tactile-Patch Encoder represents tactile observations as structured tactile patch features that preserve finger identity and local contact structure, providing contact cues for fine-grained dexterous control. Second, its high-frequency action module jointly predicts future tactile states and action chunks and refines both using incoming tactile feedback during execution. This closed-loop refinement keeps tactile predictions aligned with evolving physical interactions, enabling responsive action correction and improving robustness to contact changes and execution errors. We further introduce XHT-Dataset, comprising 900 real-world demonstrations across seven contact-rich tasks collected on an XHand--UR7e platform, and evaluate ReTouch through closed-loop real-robot experiments. ReTouch surpasses the strongest baseline by 18.4 and 23.8 percentage points in average success rate under standard and challenging conditions, respectively, demonstrating its effectiveness and robustness.
Shiqi Zhang, Xin Zhang, Yedong Shen +9
University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK
Dexterous manipulation in everyday environments requires both anticipation and reaction: a robot must predict how contact should evolve while rapidly correcting local errors caused by slip, misalignment, unstable grasping, or force mismatch. Vision and language provide semantic and geometric guidance, but they cannot reliably reveal hidden contact states such as force, slip, and contact stability. Although tactile sensing exposes these physical cues, most existing policies treat touch as a low-frequency observation stream within a monolithic action model, coupling slow task reasoning, action generation, and fast contact feedback in a single loop. We introduce TouchWorld, a predictive-and-reactive tactile foundation model for dexterous manipulation. TouchWorld uses a hierarchical policy that separates vision-language subtask planning, tactile world-model prediction, visuo-tactile goal-conditioned action generation, and high-frequency tactile residual refinement. A High-Level Planning Layer produces executable subtasks and predicts tactile subgoals; a Visuo-Tactile Goal-Conditioned Policy generates nominal action chunks; and a Tactile-Conditioned Refinement Policy performs online residual correction using recent tactile and proprioceptive feedback. By using touch as both a predictive contact reference and a fast feedback signal, TouchWorld preserves the semantic generalization of vision-language-action policies while improving local contact adaptation. Across six long-horizon and contact-rich dexterous manipulation tasks, TouchWorld achieves 65.0% success in the clean setting and 53.7% success under human perturbations, outperforming the strongest baseline by 15.7 and 18.5 percentage points, respectively.
Jianyi Zhou, Feiyang Hong, Yunhao Li +9
Harbin Institute of Technology, Shenzhen · PHANES AI
Contact-rich manipulation requires robots to continuously perceive and regulate evolving physical interactions under dynamic contact transitions or complex surface geometries. Recent imitation learning methods improve contact-aware control by incorporating tactile or force feedback, but they rarely model the asymmetric spatiotemporal roles of global force and local tactile sensing. To address this, we propose TacForeSight, a lightweight force-conditioned tactile foresight framework for real-time manipulation. The core component is TacForceWM, a tactile world model that predicts short-horizon tactile latent dynamics from dual-finger tactile observations conditioned on high-frequency wrist force and torque signals. Another key component, the Predictive Tactile-Conditioned Policy, leverages the predicted latents as anticipatory contact priors, models the current-to-future tactile evolution via cross-attention, and adaptively fuses visuo-tactile features through a tactile-guided gating module. By forecasting purely within a compact latent space, TacForeSight enables proactive contact reasoning with efficient real-time inference suitable for high-frequency manipulation control. Real-robot experiments on five representative tasks and three in-process perturbation settings show that TacForeSight consistently outperforms existing baselines, particularly under dynamic contact disturbances. All models and datasets will be made publicly available on the project website at https://tacforesight.github.io/ProjectPage.
Yujie Zang, Yuhang Zheng, Xian Nie +7
TARS Robotics · National University of Singapore · Shanghai Jiao Tong University +2