Tactile sensing plays a crucial role in everyday human interaction. Consider inserting a plug into a tight socket, vision alone provides limited guidance during the final insertion stage, and we instead rely on both anticipated contact and realized tactile feedback to ajust the insertion motion. The same capability is equally important for robots, particularly in dexterous manipulation tasks. Yet many contemporary robot policies remain vision-dominant. This visual bias, compounded by limited tactile data and sparse contact events, can cause directly incorporated tactile signals to be ignored or even compromise the policy's pretrained capabilities, yielding degrading performance as observed in our experiments. We present TF2-Steering, which refines actions via learned residual steering of the flow-matching action expert's velocity field, using a tactile mixture-of-experts architecture to integrate both tactile forecasts and real-time tactile feedback. Together with specific designed three stages training recipe, TF2-Steering enables tactile signals to correct and, when necessary, redirect the flow-matching action-generation process, while building on the strong action-generation capacity of the vision-based foundation policy. Extensive experiments across six simulated and six real-world manipulation tasks demonstrate the effectiveness of TF2-Steering. It outperforms the strongest evaluated baselines by 10.7 percentage points in average simulation success rate and 9.1 percentage points in average real-world progress success rate. Project page: https://yiming-cc.github.io/tf2-steering/.
Figures & tables
Figure 1: Overview of TF2 -Steering. Left: visual-only and visual–tactile pretraining data. Middle: the tactile expert uses cached forecasts from the tactile world model and real-time tactile feedback to predict delta velocity, refining the action expert’s flow-matching velocity. The video world model and action expert remain independently usable without tactile input. Right: task-specific post-training and evaluation in simulation and the real world.
Figure 2: Overview of TF2 -Steering. The video and tactile world models provide low-frequency planning priors, while the action and tactile experts perform high-frequency action refinement through learned delta-velocity corrections. Unidirectional attention keeps the original world-action backbone independent of tactile input. (a) Asynchronous inference uses incoming tactile feedback to refine subsequent action blocks, with pre-execution tactile forecasts held fixed throughout each block. (b) The tactile expert integrates predicted and real-time tactile features through separate subexperts with sigmoid weighting to steer the action expert’s flow-matching velocity field.
Figure 3: Real-world manipulation tasks and example execution sequences. (a) Bottle-cap twisting and zipper closing, with tactile visualizations shown as insets. (b) Task scenes for fruit placement, whiteboard wiping, cup separation, liquid transfer with a dropper, bottle-cap twisting, and zipper closing.
Method
Lift Bottle
Pull Out Key
Lift Can
Put Bottle
Insert Hole
Insert Tube
Avg.
ACT ∗ ( Zhao et al., 2023 )
71.0
46.0
29.0
31.0
24.0
56.0
42.8
VITaL ∗ ( George et al., 2025 )
72.0
47.0
8.0
32.0
25.0
34.0
36.3
FTP-1 ( Yuan et al., 2026a )
88.0
14.0
68.0
74.0
26.0
76.0
57.7
π0.5 ( Physical Intelligence et al., 2025 )
94.0
36.0
74.0
6.0
56.0
68.0
55.7
π0.5 + tactile
96.0
38.0
68.0
14.0
48.0
72.0
56.0
TF2 -Steering (ours)
98.0
34.0
60.0
44.0
94.0
80.0
68.3
Table 1: Simulation results on UniVTAC. Success rate (%, ↑ ) on six tasks; Avg. denotes the macro-average. Bold and underlined values indicate the best and second-best results in each column, respectively, including ties. ∗ ACT and VITaL results are taken from UniVTAC ( Chen et al., 2026a ) .
Method
Fruit on Plate
Close Zipper
Wipe Whiteboard
Twist Cap
Split Cups
Dropper Transfer
Avg.
ViTacFormer ( Heng et al., 2025 )
12.5
4.0
2.0
9.0
7.0
2.0
6.1
π0.5 ( Physical Intelligence et al., 2025 )
60.0
18.0
36.0
26.0
62.0
39.0
40.2
π0.5 + tactile
58.0
12.0
39.0
22.0
57.0
23.0
35.2
T-Rex ( Niu et al., 2026 )
72.5
56.0
61.0
58.0
87.0
40.0
62.4
FTP-1 ( Yuan et al., 2026a )
75.0
36.0
55.0
40.0
64.0
46.0
52.7
TF2 -Steering (ours)
70.0
60.0
76.0
67.0
90.0
66.0
71.5
Table 2: Real-world results. Progress success rate (%, ↑ ) over 20 trials per task; Avg. denotes the macro-average over the six tasks. Bold and underlined values indicate the best and second-best results in each column, respectively.
Configuration
Twist Cap
Close Zipper
Split Cups
Avg.
Δ Avg. (pp)
TF2 -Steering (full model)
67.0
60.0
90.0
72.3
–
Tactile modalities
w/o predicted tactile
61.0
58.0
90.0
69.7
−2.7
w/o real-time tactile
54.0
49.0
86.0
63.0
−9.3
w/o both tactile modalities
51.0
50.0
84.0
61.7
−10.7
Tactile integration strategy
Table 3: Model design ablations. Real-world progress success rate (%, ↑ ) on three manipulation tasks; Avg. denotes the macro-average. Δ Avg. reports the change relative to the full model in percentage points (pp), computed before rounding; negative values indicate a decline. Bold indicates the best result in each task and average column, including ties.
Figure 4: Effect of tactile pretraining. (a) Real-world progress success rates with and without tactile pretraining on four manipulation tasks. (b) Qualitative comparison of tactile forecasts. From left to right: visual observation, ground-truth tactile observation, and tactile predictions from models with and without tactile pretraining.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Inputs and architecture
Image resolution ( H×W )
192×256
Future video frames / target frame rate
24 / 15 fps
Action horizon
50 steps
Action / state dimension
128 / 128
Video transformer (layers / heads / head dimension)
28 / 32 / 64
Appendix
Table 4: World-action backbone pretraining configuration. The training run uses 100,000 optimizer steps and a global batch size of 512 . The action horizon and delay settings refer to backbone pretraining.
Indices
Dimension
Assigned quantity
0 – 9
10
Left-arm joint representation
10 – 19
10
Left TCP / end-effector representation
20
1
Left gripper
21 – 49
29
Left dexterous hand
50 – 59
10
Right-arm joint representation
60 – 69
10
Right TCP / end-effector representation
Appendix
Table 5: Canonical 128-dimensional action and state layout. Indices are zero-based and inclusive. Group dimensions indicate slot capacity, rather than the number of active degrees of freedom for every embodiment. TCP denotes the tool center point.
Setting
Stage 1
Stage 2
Training steps
50,000
50,000
Global batch size
1,024
512
Tactile / tactile-free sampling
100% / 0%
90% / 10%
Video model and Q-Former learning rate
–
10−5
Tactile world model learning rate
10−4
5×10−5
Action expert learning rate
–
5×10−5
Appendix
Table 6: Key settings for the two tactile pretraining stages. Stage 1 trains only the tactile world model; Stage 2 jointly trains all four modules and the Q-Former. Batch sizes are global; dashes indicate inapplicable settings.
Figure 5: Real-robot platform and demonstration collection. (a) Two UR5e arms equipped with Sharpa dexterous hands share a tabletop workspace. A top-mounted Intel RealSense D455 and two wrist-mounted D405 cameras provide complementary visual observations. (b) Meta Quest supplies arm pose commands, while Manus gloves control hand articulation. Demonstrations are collected at 30 Hz.
Task
Progress milestones and scores
Fruit on Plate
0.25 for each object successfully placed on the plate; 1.0 when all four objects are placed.
Close Zipper
0.2 : the left hand lifts the bag; 0.6 : the right hand begins pulling the zipper, which is not yet fully closed; 1.0 : the zipper is fully closed.
Wipe Whiteboard
0.2 : the robot grasps the eraser; 0.6 : it performs an initial wiping stroke on the board; 1.0 : the board is wiped clean.
Twist Cap
0.2 : the left hand lifts the bottle; 0.6 : the robot successfully rotates the cap; 1.0 : the cap is fully unscrewed.
Split Cups
0.2 : the robot lifts the cups; 0.6 : it inserts its fingers into a cup; 1.0 : one cup is separated from the others.
Dropper Transfer
0.2 : the robot grasps the dropper; 0.8 : liquid is transferred into the beaker; 1.0 : the dropper is put back after the transfer.
Appendix
Table 7: Progress scoring for real-world evaluation. Scores range from 0 to 1 for each trial. Sequential-task scores indicate cumulative progress, rather than additive rewards.
Figure 6: Representative real-world policy rollouts. (a) Split Cups. (b) Wipe Whiteboard. Each row shows five selected frames in chronological order from left to right, with the task prompt at the upper left.