Tactile sensing plays a crucial role in everyday human interaction. Consider inserting a plug into a tight socket, vision alone provides limited guidance during the final insertion stage, and we instead rely on both anticipated contact and realized tactile feedback to ajust the insertion motion. The same capability is equally important for robots, particularly in dexterous manipulation tasks. Yet many contemporary robot policies remain vision-dominant. This visual bias, compounded by limited tactile data and sparse contact events, can cause directly incorporated tactile signals to be ignored or even compromise the policy's pretrained capabilities, yielding degrading performance as observed in our experiments. We present TF2-Steering, which refines actions via learned residual steering of the flow-matching action expert's velocity field, using a tactile mixture-of-experts architecture to integrate both tactile forecasts and real-time tactile feedback. Together with specific designed three stages training recipe, TF2-Steering enables tactile signals to correct and, when necessary, redirect the flow-matching action-generation process, while building on the strong action-generation capacity of the vision-based foundation policy. Extensive experiments across six simulated and six real-world manipulation tasks demonstrate the effectiveness of TF2-Steering. It outperforms the strongest evaluated baselines by 10.7 percentage points in average simulation success rate and 9.1 percentage points in average real-world progress success rate. Project page: https://yiming-cc.github.io/tf2-steering/.
Figures & tables
Figure 1: Overview of TF2 -Steering. Left: visual-only and visual–tactile pretraining data. Middle: the tactile expert uses cached forecasts from the tactile world model and real-time tactile feedback to predict delta velocity, refining the action expert’s flow-matching velocity. The video world model and action expert remain independently usable without tactile input. Right: task-specific post-training and evaluation in simulation and the real world.
Figure 2: Overview of TF2 -Steering. The video and tactile world models provide low-frequency planning priors, while the action and tactile experts perform high-frequency action refinement through learned delta-velocity corrections. Unidirectional attention keeps the original world-action backbone independent of tactile input. (a) Asynchronous inference uses incoming tactile feedback to refine subsequent action blocks, with pre-execution tactile forecasts held fixed throughout each block. (b) The tactile expert integrates predicted and real-time tactile features through separate subexperts with sigmoid weighting to steer the action expert’s flow-matching velocity field.
Figure 3: Real-world manipulation tasks and example execution sequences. (a) Bottle-cap twisting and zipper closing, with tactile visualizations shown as insets. (b) Task scenes for fruit placement, whiteboard wiping, cup separation, liquid transfer with a dropper, bottle-cap twisting, and zipper closing.
Method
Lift Bottle
Pull Out Key
Lift Can
Put Bottle
Insert Hole
Insert Tube
Avg.
ACT ∗ ( Zhao et al., 2023 )
71.0
46.0
29.0
31.0
24.0
56.0
42.8
VITaL ∗ ( George et al., 2025 )
72.0
47.0
8.0
32.0
25.0
34.0
36.3
FTP-1 ( Yuan et al., 2026a )
88.0
14.0
68.0
74.0
26.0
76.0
57.7
π0.5 ( Physical Intelligence et al., 2025 )
94.0
36.0
74.0
6.0
56.0
68.0
55.7
π0.5 + tactile
96.0
38.0
68.0
14.0
48.0
72.0
56.0
TF2 -Steering (ours)
98.0
34.0
60.0
44.0
94.0
80.0
68.3
Table 1: Simulation results on UniVTAC. Success rate (%, ↑ ) on six tasks; Avg. denotes the macro-average. Bold and underlined values indicate the best and second-best results in each column, respectively, including ties. ∗ ACT and VITaL results are taken from UniVTAC ( Chen et al., 2026a ) .
Method
Fruit on Plate
Close Zipper
Wipe Whiteboard
Twist Cap
Split Cups
Dropper Transfer
Avg.
ViTacFormer ( Heng et al., 2025 )
12.5
4.0
2.0
9.0
7.0
2.0
6.1
π0.5 ( Physical Intelligence et al., 2025 )
60.0
18.0
36.0
26.0
62.0
39.0
40.2
π0.5 + tactile
58.0
12.0
39.0
22.0
57.0
23.0
35.2
T-Rex ( Niu et al., 2026 )
72.5
56.0
61.0
58.0
87.0
40.0
62.4
FTP-1 ( Yuan et al., 2026a )
75.0
36.0
55.0
40.0
64.0
46.0
52.7
TF2 -Steering (ours)
70.0
60.0
76.0
67.0
90.0
66.0
71.5
Table 2: Real-world results. Progress success rate (%, ↑ ) over 20 trials per task; Avg. denotes the macro-average over the six tasks. Bold and underlined values indicate the best and second-best results in each column, respectively.
Configuration
Twist Cap
Close Zipper
Split Cups
Avg.
Δ Avg. (pp)
TF2 -Steering (full model)
67.0
60.0
90.0
72.3
–
Tactile modalities
w/o predicted tactile
61.0
58.0
90.0
69.7
−2.7
w/o real-time tactile
54.0
49.0
86.0
63.0
−9.3
w/o both tactile modalities
51.0
50.0
84.0
61.7
−10.7
Tactile integration strategy
Table 3: Model design ablations. Real-world progress success rate (%, ↑ ) on three manipulation tasks; Avg. denotes the macro-average. Δ Avg. reports the change relative to the full model in percentage points (pp), computed before rounding; negative values indicate a decline. Bold indicates the best result in each task and average column, including ties.
Figure 4: Effect of tactile pretraining. (a) Real-world progress success rates with and without tactile pretraining on four manipulation tasks. (b) Qualitative comparison of tactile forecasts. From left to right: visual observation, ground-truth tactile observation, and tactile predictions from models with and without tactile pretraining.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Inputs and architecture
Image resolution ( H×W )
192×256
Future video frames / target frame rate
24 / 15 fps
Action horizon
50 steps
Action / state dimension
128 / 128
Video transformer (layers / heads / head dimension)
28 / 32 / 64
Appendix
Table 4: World-action backbone pretraining configuration. The training run uses 100,000 optimizer steps and a global batch size of 512 . The action horizon and delay settings refer to backbone pretraining.
Indices
Dimension
Assigned quantity
0 – 9
10
Left-arm joint representation
10 – 19
10
Left TCP / end-effector representation
20
1
Left gripper
21 – 49
29
Left dexterous hand
50 – 59
10
Right-arm joint representation
60 – 69
10
Right TCP / end-effector representation
Appendix
Table 5: Canonical 128-dimensional action and state layout. Indices are zero-based and inclusive. Group dimensions indicate slot capacity, rather than the number of active degrees of freedom for every embodiment. TCP denotes the tool center point.
Setting
Stage 1
Stage 2
Training steps
50,000
50,000
Global batch size
1,024
512
Tactile / tactile-free sampling
100% / 0%
90% / 10%
Video model and Q-Former learning rate
–
10−5
Tactile world model learning rate
10−4
5×10−5
Action expert learning rate
–
5×10−5
Appendix
Table 6: Key settings for the two tactile pretraining stages. Stage 1 trains only the tactile world model; Stage 2 jointly trains all four modules and the Q-Former. Batch sizes are global; dashes indicate inapplicable settings.
Figure 5: Real-robot platform and demonstration collection. (a) Two UR5e arms equipped with Sharpa dexterous hands share a tabletop workspace. A top-mounted Intel RealSense D455 and two wrist-mounted D405 cameras provide complementary visual observations. (b) Meta Quest supplies arm pose commands, while Manus gloves control hand articulation. Demonstrations are collected at 30 Hz.
Task
Progress milestones and scores
Fruit on Plate
0.25 for each object successfully placed on the plate; 1.0 when all four objects are placed.
Close Zipper
0.2 : the left hand lifts the bag; 0.6 : the right hand begins pulling the zipper, which is not yet fully closed; 1.0 : the zipper is fully closed.
Wipe Whiteboard
0.2 : the robot grasps the eraser; 0.6 : it performs an initial wiping stroke on the board; 1.0 : the board is wiped clean.
Twist Cap
0.2 : the left hand lifts the bottle; 0.6 : the robot successfully rotates the cap; 1.0 : the cap is fully unscrewed.
Split Cups
0.2 : the robot lifts the cups; 0.6 : it inserts its fingers into a cup; 1.0 : one cup is separated from the others.
Dropper Transfer
0.2 : the robot grasps the dropper; 0.8 : liquid is transferred into the beaker; 1.0 : the dropper is put back after the transfer.
Appendix
Table 7: Progress scoring for real-world evaluation. Scores range from 0 to 1 for each trial. Sequential-task scores indicate cumulative progress, rather than additive rewards.
Figure 6: Representative real-world policy rollouts. (a) Split Cups. (b) Wipe Whiteboard. Each row shows five selected frames in chronological order from left to right, with the task prompt at the upper left.
Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution. While prior methods typically perform this verification only with visual observations, vision alone is often insufficient for contact-rich manipulation, where success depends on both global task progress and subtle local interactions such as contact force. We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem. At the high level, visual sampling-and-verification performs long-horizon mode selection, deciding what behavior the robot should execute. At the low level, tactile-guided diffusion editing refines the selected action sequence over a shorter horizon to satisfy local contact requirements. To support outcome-based steering, ViTaL learns a visuo-tactile latent world model and employs semantically aligned visual and tactile verifiers, including a novel text-conditioned tactile reward that scores predicted tactile futures directly in latent space. Across three real-world contact-rich manipulation tasks, ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%. Website: https://yilin-wu98.github.io/vital_website.
Contact-rich manipulation requires adapting to contact states that can evolve substantially within an action horizon. However, chunk-based vision-language-action models predict complete action chunks from observations collected before execution, leaving tactile conditioning stale during execution. Existing tactile-reactive approaches typically rely on separate high-frequency controllers, which increase both architectural and training complexity. In this paper, we introduce TacForcing, a streaming action-generation framework incorporating execution-time tactile feedback. TacForcing replaces the standard action expert with a streaming expert that generates action blocks sequentially while preserving intermediate states of unfinished blocks. After each block is executed, the expert resumes generation from these states using newly acquired tactile feedback. To better align tactile conditioning with action execution, we further introduce Execution-Aware Tactile Attention (EATA), which restricts direct access to each tactile update to the next block scheduled for execution. Across six UniVTAC simulation tasks and six real-world contact-rich manipulation tasks on two robot platforms, TacForcing achieves average success rates of 65% in simulation and 66% in the real world, outperforming the strongest baselines by 6 and 15 percentage points, respectively.
Jianbo Zhou, Boyuan Zhao, Yuzheng Zhang +9
School of Computer Science, SJTU · SCUT · ECUST +5
Fusing tactile signals has proven effective for contact-rich manipulation, enabling robots to perceive contact states and adapt to rapidly changing physical interactions. Yet effectively integrating tactile feedback into dexterous manipulation remains underexplored. In this work, we introduce ReTouch, a vision-language-action model (VLA) that supports contact-rich dexterous manipulation through tactile predictions continually refined online using execution-time feedback. ReTouch builds on two main innovations for tactile representation and closed-loop action generation. First, its Tactile-Patch Encoder represents tactile observations as structured tactile patch features that preserve finger identity and local contact structure, providing contact cues for fine-grained dexterous control. Second, its high-frequency action module jointly predicts future tactile states and action chunks and refines both using incoming tactile feedback during execution. This closed-loop refinement keeps tactile predictions aligned with evolving physical interactions, enabling responsive action correction and improving robustness to contact changes and execution errors. We further introduce XHT-Dataset, comprising 900 real-world demonstrations across seven contact-rich tasks collected on an XHand--UR7e platform, and evaluate ReTouch through closed-loop real-robot experiments. ReTouch surpasses the strongest baseline by 18.4 and 23.8 percentage points in average success rate under standard and challenging conditions, respectively, demonstrating its effectiveness and robustness.
Shiqi Zhang, Xin Zhang, Yedong Shen +9
University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK