Contact-rich manipulation requires robots to coordinate not only where to move, but also how contact and force should evolve throughout an interaction. Yet most robot policies represent actions primarily as motion trajectories, using tactile signals as inputs or auxiliary prediction targets without explicitly encoding contact dynamics in the action representation. We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion, contact, and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement. We learn these latents through visually conditioned joint action-tactile reconstruction with a time-conditioned implicit decoder. Using them as flow-matching generation targets supports post-training of pretrained vision-language-action models (VLAs) and world action models (WAMs), as well as training flow-matching policies from scratch. A shared decoder maps generated latents to executable actions. For offline improvement, we learn a tactile-conditioned expert reference model from interaction latents encoded from expert demonstrations. The resulting expert-consistency rewards guide weighted learning from successful rollouts, while observation-compatible expert references guide correction of low-reward predictions from failed rollouts. Across three policy families, ILA improves success rates averaged over four real-world tasks by 21.0-36.5 percentage points relative to tactile-conditioned direct-action baselines. Ablations highlight the benefit of encoding contact evolution in action targets beyond tactile conditioning, while the same latent space supports expert-guided offline improvement. Project website: https://mrsecant.github.io/ILA/.
Figures & tables
Figure 1: Interaction Latents as Actions (ILA). A shared latent action jointly represents motion and tactile evolution over temporally extended interactions and decodes into executable actions. Policies across architectures generate these latents, while expert-consistency rewards and compatible references guide offline improvement, unifying action generation, evaluation, and correction.
Figure 2: Overview of our proposed ILA for policy learning and offline improvement.
Figure 3: Experimental setup and contact-rich manipulation tasks. Left: dual-arm robotic platform equipped with wrist and base cameras and fingertip tactile sensors. Right: real-world manipulation tasks requiring diverse patterns of motion—contact coordination.
Cucumber Peeling
Egg Flipping
Celery Cutting
Plate Wiping
Family
Method
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
Interaction latent actions
FM
FM-tac
50%
2.65±0.44
85%
1.25±0.84
16%
10.48±0.51
65%
5.51±0.31
FM-tac ∗
15%
2.65±0.49
65%
0.79±0.76
14%
6.06±1.57
80%
2.87±0.50
FM-ILA
75%
2.04±0.37
95%
0.56±0.64
50%
7.52±0.78
100%
3.44±0.35
VLA ( π0.5 )
VLA-tac
15%
2.58±0.43
25%
2.27±1.00
6%
7.82±1.56
75%
2.69±0.53
Table 1: Performance on four contact-rich manipulation tasks. We report task success rate (SR) and execution force-proxy error ( Fext L1, mean ± standard deviation) relative to expert references.
Figure 4: Motion–contact representation analysis on cucumber peeling. (a,b) Latent responses to action and tactile scaling; (c,d) responses to Gaussian input noise; (e,f) matched linear probes for action and force-proxy magnitude. Points denote latent states and lines trace one trajectory. Markers show mean L1 errors, with error bars indicating 95% trajectory-bootstrap intervals.
Setting
SR ↑
Fext L1 ↓
Reward alternatives
Grasp-intensity
70%
1.78±0.48
HapticVLA-style
70%
1.69±0.58
w/o failure correction
80%
2.89±0.43
Ours
85%
0.51±0.50
Table 2: Reward and correction ablations on cucumber peeling.
Figure 5: Interaction rewards during manipulation. Solid and dashed curves denote failed and successful rollouts, respectively. Reward decreases coincide with local interaction failures (red regions), while differences are smaller during phases with little interaction demand (green regions).
Cucumber Peeling
Celery Cutting
Family
Setting
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
FM
Full ILA
75%
2.04±0.37
50%
7.52±0.78
w/o implicit decoding
65%
2.32±0.53
44%
7.97±2.45
VLA ( π0.5 )
Full ILA
50%
2.28±0.41
37%
7.22±1.17
w/o S&D
40%
3.13±0.58
32%
6.51±0.92
w/o S&D + encoder vision
30%
2.78±0.46
30%
8.68±1.11
Table 3: Ablations of interaction representation, policy conditioning, and action decoding. Results are reported on cucumber peeling and celery cutting. All VLA variants retain native policy vision and tactile conditioning.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
S&D tokens
Generation target
P-tac
No
Motor actions
P-tac ∗
No
Motor actions and tactile fields
P-ILA
Yes
Interaction latents
Appendix
Table 4: Configurations of the VLA and FM policy variants.
Figure 6: Real-world task procedures. Frames in each row are drawn from a single demonstration and ordered chronologically from left to right. Cucumber peeling comprises three successive peel–place cycles, shown as six alternating stages. The other rows illustrate tool preparation, object interaction, and the resulting state for flipping, cutting, and wiping. Timestamps are relative to the beginning of each recorded episode.
Method
Key implementation settings
IQL
Twin Q-functions and an expectile value function with τ=0.7 . Terminal success receives +1 ; all other rewards are zero. The discount is 0.99Δ/128 for a transition of Δ physical frames. The Gaussian actor has two width-256 Transformer decoder layers, eight heads, and 32 latent queries; Q/V networks have two width-512 hidden layers. Actor log-likelihood weights are min{exp(3A),100} , where A=min(Q1,Q2)−V . Inference uses the conditional mean. Additional training comprises 5,000 success-only BC updates and 5,000 critic updates, followed by one critic update per actor-improvement update.
RECAP
A flow-matching actor uses positive/negative quality conditioning, with 30% condition dropout and a training-calibrated target positive fraction of 40%. A distributional value model with 201 bins on [−1,0] is fitted to Monte Carlo returns for 5,000 updates. Advantage estimates use one-chunk lookahead. Inference uses positive conditioning with guidance scale 1.
Appendix
Table 5: Offline RL baseline adaptations. Each baseline receives 2,000 actor-improvement updates; additional BC and critic/value fitting are specified below.
Figure 7: Motion–contact representation analysis across additional tasks. Rows show egg flipping, plate wiping, and celery cutting. Columns show action and tactile scaling, followed by Gaussian noise in normalized action and tactile inputs; other inputs remain fixed. Colored paths connect consecutive chunks of one held-out trajectory per task; faint points show all test chunks under the displayed conditions. Scaling and noise use separate t-SNE projections within each task.
Figure 8: Qualitative visualization of interaction representations. Two-dimensional UMAP projections for cucumber peeling, egg flipping, plate wiping, and celery cutting. Green and red denote representations from successful and failed episodes, respectively. Lines connect representations in temporal order. Shared trajectory regions and local deviations illustrate variation across recorded interactions; apparent distances and separation should be interpreted qualitatively.
Cucumber Peeling
Egg Flipping
Celery Cutting
Plate Wiping
Average
Family
Method
Tac. L1 ↓
Fpred L1 ↓
Tac. L1 ↓
Fpred L1 ↓
Tac. L1 ↓
Fpred L1 ↓
Tac. L1 ↓
Fpred L1 ↓
Tac. L1 ↓
Fpred L1 ↓
FM
FM-tac ∗
0.0125
3.07
0.0130
4.16
0.0083
1.93
0.0116
2.61
0.0114
2.94
FM-ILA
0.0124
2.70
0.0100
2.57
0.0068
1.21
0.0105
1.48
0.0099
1.99
VLA ( π0.5 )
VLA-tac ∗
0.0114
2.81
0.0128
3.45
0.0084
2.26
0.0111
2.69
0.0109
2.80
VLA-ILA
0.0122
2.63
0.0101
2.45
0.0070
1.26
0.0103
1.37
0.0099
1.93
WAM
DreamTac
0.0094
2.72
0.0114
3.29
0.0061
1.59
0.0085
2.11
0.0088
2.43
Appendix
Table 6: Open-loop tactile and force-proxy prediction. We report mean Tac. L1 and Fpred L1; Average is the unweighted mean across the four tasks.
Cucumber Peeling
Egg Flipping
Celery Cutting
Plate Wiping
Method
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
SR ↑
Fext L1 ↓
Action-trajectory replay
40%
1.11±0.55
100%
1.49±0.79
46%
6.54±3.30
50%
4.05±0.32
Imitation-trained ILA policies
FM-ILA
75%
2.04±0.37
95%
0.56±0.64
50%
7.52±0.78
100%
3.44±0.35
VLA-ILA
50%
2.28±0.41
85%
0.38±0.54
37%
7.22±1.17
95%
2.51±0.38
WAM-ILA
70%
1.44±0.47
80%
0.14±0.50
45%
8.94±0.63
100%
1.33±0.49
Appendix
Table 7: ILA policies and action-trajectory replay. We report SR and Fext L1 (mean ± standard deviation).
Figure 9: Successful-window reweighting across four tasks. Left: empirical weight distributions. Right: mean weight within each reward decile. Dashed lines indicate uniform weighting, wi=1 . Weights are recomputed from cached rewards and the fixed calibration of each completed run, using η=0.3 , β=2 , and aclip=3 . The diagnostic pools contain successful-rollout policy predictions.
Figure 10: Observation-compatible pairs across four tasks. Failure and reference data occupy the upper and lower rows of each task. Wrist images and tactile maps are taken at the respective window starts. The four tactile maps show local tangential deformation magnitudes, with a shared color scale within each task. The tactile curve sums the four per-sensor spatial XY resultant magnitudes in raw deformation units. TCP shows the mean positional displacement of the two recorded end effectors from their respective window starts.
Figure 11: Four tasks encoded by a shared Stage-1 model. Each point represents a 128-dimensional token and is colored by task. All tasks are encoded by the same model with shared preprocessing.
Task
Model
Action ↓
Tactile ↓
Force ↓
×10−3
×10−3
×10−2
Cucumber peeling
Task-specific
34.21
3.50
25.80
Shared
22.63
3.51
23.72
Egg flipping
Task-specific
18.05
2.39
25.37
Shared
6.12
2.87
16.54
Celery cutting
Task-specific
6.90
1.21
14.25
Appendix
Table 8: Shared versus task-specific reconstruction. Action and tactile report L1 errors in the original recorded coordinates. Force reports magnitude L1 of the directly decoded aggregate deformation, a force proxy in sensor-recording units rather than newtons.
Figure 12: Successful task interactions across four tasks. The sequences show strip formation and transfer, egg turnover, plate wiping, and celery separation and transfer. The peeling row depicts one completed peel–place cycle, rather than all three cycles required for rollout success. Cutting is assessed per cut; the displayed separated pieces and transfer do not imply that every preceding cut succeeded.
Task
Required motion–contact coordination
Cases
Peeling
Maintain a productive stroke; acquire and transfer the strip.
P1–P4
Flipping
Support, turn, and release the egg while retaining the tool.
E1–E5
Wiping
Retain both objects and move effective contact across the marks.
W1–W3
Cutting
Stabilize, sever, and clear the knife between strokes.
C1–C5
Appendix
Table 9: Interaction requirements illustrated by the rollout cases. Case IDs correspond to the rows of Figs. 13 – 15 .
Figure 13: Failure cases in cucumber peeling and plate wiping. P1–P4 show missed acquisition, strip breakage, premature lifting, and tool overrun. W1–W3 show an unstable sponge grasp, residual staining despite a retained grasp, and prolonged wiping with limited coverage. W2–W3 end with enlarged views of residual markings. The images do not determine contact pressure.
Figure 14: Failure cases in egg flipping. The rows distinguish attachment during rotation (E1), missing or inadequate support (E2–E3), withdrawal without turnover (E4), and premature tool release (E5). Frames retain the observed tool and object states; they do not isolate insertion depth or lifting clearance as a unique cause of failure.
Figure 15: Failure cases in celery cutting. C1–C5 illustrate incomplete separation, inadequate blade clearance, loss of stabilization, and unstable acquisition. A cut succeeds only if the celery is completely severed at the cutting location before the knife clears the stalk. Residual connections, including cases where the partially cut stalk follows the lifted knife, count as failures. Rows show selected failure phases; individual cuts within the same recording may have different outcomes.
Tactile-enhanced vision-language-action (VLA) policies have been introduced for contact-rich manipulation, where critical interaction states are often hidden from vision. Future tactile prediction is a promising way to use touch because it turns tactile outcomes into supervision for action-induced contact dynamics. Yet VLA policies contain representations with different roles, from perceptual encoding to motor prediction, making it unclear where this supervision should be applied. We study this as a representation-alignment problem. Through a linear probe analysis, we find that future tactile states are most predictable from intermediate action-expert features, rather than from vision-language features or final action states. Motivated by this observation, we introduce a lightweight Latent Tactile Predictor (LTP), which predicts compact future tactile embeddings from the identified intermediate representation. By avoiding direct prediction of noisy raw tactile signals, LTP provides an action-outcome grounding signal that aligns intermediate action representations with future contact consequences. Experiments on real-world contact-rich manipulation tasks show that representation-aligned tactile grounding outperforms less aligned or multi-interface tactile prediction, highlighting the importance of where tactile supervision is applied.
Ruilin Chen, Jingkai Jia, Tong Yang +8
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · TeleAl, China Telecom +2
Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result videos and more details are available at https://tac-corl.github.io/
Siyu Ma, Yuqi Liang, Chang Yu +5
University of California, Los Angeles · University of California, San Diego · 4Peking University +1
Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.