cs.ROOct 13, 2026
SaveInteraction Latents as Actions for Contact-Rich Manipulation
Organizations: NUS · Anyverse Robotics · BIT · Open-Phoenix Team · UCAS · FDU
Abstract
Contact-rich manipulation requires robots to coordinate not only where to move, but also how contact and force should evolve throughout an interaction. Yet most robot policies represent actions primarily as motion trajectories, using tactile signals as inputs or auxiliary prediction targets without explicitly encoding contact dynamics in the action representation. We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion, contact, and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement. We learn these latents through visually conditioned joint action-tactile reconstruction with a time-conditioned implicit decoder. Using them as flow-matching generation targets supports post-training of pretrained vision-language-action models (VLAs) and world action models (WAMs), as well as training flow-matching policies from scratch. A shared decoder maps generated latents to executable actions. For offline improvement, we learn a tactile-conditioned expert reference model from interaction latents encoded from expert demonstrations. The resulting expert-consistency rewards guide weighted learning from successful rollouts, while observation-compatible expert references guide correction of low-reward predictions from failed rollouts. Across three policy families, ILA improves success rates averaged over four real-world tasks by 21.0-36.5 percentage points relative to tactile-conditioned direct-action baselines. Ablations highlight the benefit of encoding contact evolution in action targets beyond tactile conditioning, while the same latent space supports expert-guided offline improvement. Project website: https://mrsecant.github.io/ILA/.
Figures & tables
Figure 1: Interaction Latents as Actions (ILA). A shared latent action jointly represents motion and tactile evolution over temporally extended interactions and decodes into executable actions. Policies across architectures generate these latents, while expert-consistency rewards and compatible references guide offline improvement, unifying action generation, evaluation, and correction.
Figure 2: Overview of our proposed ILA for policy learning and offline improvement.
Figure 3: Experimental setup and contact-rich manipulation tasks. Left: dual-arm robotic platform equipped with wrist and base cameras and fingertip tactile sensors. Right: real-world manipulation tasks requiring diverse patterns of motion—contact coordination.
| Cucumber Peeling | Egg Flipping | Celery Cutting | Plate Wiping | ||||||
| Family | Method | SR | L1 | SR | L1 | SR | L1 | SR | L1 |
| Interaction latent actions | |||||||||
| FM | FM-tac | 50% | 85% | 16% | 65% | ||||
| FM-tac ∗ | 15% | 65% | 14% | 80% | |||||
| FM-ILA | 75% | 95% | 50% | 100% | |||||
| VLA ( ) | VLA-tac | 15% | 25% | 6% | 75% | ||||
Table 1: Performance on four contact-rich manipulation tasks. We report task success rate (SR) and execution force-proxy error ( L1, mean standard deviation) relative to expert references.
Figure 4: Motion–contact representation analysis on cucumber peeling. (a,b) Latent responses to action and tactile scaling; (c,d) responses to Gaussian input noise; (e,f) matched linear probes for action and force-proxy magnitude. Points denote latent states and lines trace one trajectory. Markers show mean L1 errors, with error bars indicating 95% trajectory-bootstrap intervals.
| Setting | SR | L1 |
|---|---|---|
| Reward alternatives | ||
| Grasp-intensity | 70% | |
| HapticVLA-style | 70% | |
| w/o failure correction | 80% | |
| Ours | 85% | |
Table 2: Reward and correction ablations on cucumber peeling.
Figure 5: Interaction rewards during manipulation. Solid and dashed curves denote failed and successful rollouts, respectively. Reward decreases coincide with local interaction failures (red regions), while differences are smaller during phases with little interaction demand (green regions).
| Cucumber Peeling | Celery Cutting | ||||
| Family | Setting | SR | L1 | SR | L1 |
| FM | Full ILA | 75% | 50% | ||
| w/o implicit decoding | 65% | 44% | |||
| VLA ( ) | Full ILA | 50% | 37% | ||
| w/o S&D | 40% | 32% | |||
| w/o S&D + encoder vision | 30% | 30% | |||
Table 3: Ablations of interaction representation, policy conditioning, and action decoding. Results are reported on cucumber peeling and celery cutting. All VLA variants retain native policy vision and tactile conditioning.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | S&D tokens | Generation target |
|---|---|---|
| P-tac | No | Motor actions |
| P-tac ∗ | No | Motor actions and tactile fields |
| P-ILA | Yes | Interaction latents |
Table 4: Configurations of the VLA and FM policy variants.
Figure 6: Real-world task procedures. Frames in each row are drawn from a single demonstration and ordered chronologically from left to right. Cucumber peeling comprises three successive peel–place cycles, shown as six alternating stages. The other rows illustrate tool preparation, object interaction, and the resulting state for flipping, cutting, and wiping. Timestamps are relative to the beginning of each recorded episode.
| Method | Key implementation settings |
|---|---|
| IQL | Twin Q-functions and an expectile value function with . Terminal success receives ; all other rewards are zero. The discount is for a transition of physical frames. The Gaussian actor has two width-256 Transformer decoder layers, eight heads, and 32 latent queries; Q/V networks have two width-512 hidden layers. Actor log-likelihood weights are , where . Inference uses the conditional mean. Additional training comprises 5,000 success-only BC updates and 5,000 critic updates, followed by one critic update per actor-improvement update. |
| RECAP | A flow-matching actor uses positive/negative quality conditioning, with 30% condition dropout and a training-calibrated target positive fraction of 40%. A distributional value model with 201 bins on is fitted to Monte Carlo returns for 5,000 updates. Advantage estimates use one-chunk lookahead. Inference uses positive conditioning with guidance scale 1. |
Table 5: Offline RL baseline adaptations. Each baseline receives 2,000 actor-improvement updates; additional BC and critic/value fitting are specified below.
Figure 7: Motion–contact representation analysis across additional tasks. Rows show egg flipping, plate wiping, and celery cutting. Columns show action and tactile scaling, followed by Gaussian noise in normalized action and tactile inputs; other inputs remain fixed. Colored paths connect consecutive chunks of one held-out trajectory per task; faint points show all test chunks under the displayed conditions. Scaling and noise use separate t-SNE projections within each task.
Figure 8: Qualitative visualization of interaction representations. Two-dimensional UMAP projections for cucumber peeling, egg flipping, plate wiping, and celery cutting. Green and red denote representations from successful and failed episodes, respectively. Lines connect representations in temporal order. Shared trajectory regions and local deviations illustrate variation across recorded interactions; apparent distances and separation should be interpreted qualitatively.
| Cucumber Peeling | Egg Flipping | Celery Cutting | Plate Wiping | Average | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Family | Method | Tac. L1 | L1 | Tac. L1 | L1 | Tac. L1 | L1 | Tac. L1 | L1 | Tac. L1 | L1 |
| FM | FM-tac ∗ | ||||||||||
| FM-ILA | |||||||||||
| VLA ( ) | VLA-tac ∗ | ||||||||||
| VLA-ILA | |||||||||||
| WAM | DreamTac | ||||||||||
Table 6: Open-loop tactile and force-proxy prediction. We report mean Tac. L1 and L1; Average is the unweighted mean across the four tasks.
| Cucumber Peeling | Egg Flipping | Celery Cutting | Plate Wiping | |||||
| Method | SR | L1 | SR | L1 | SR | L1 | SR | L1 |
| Action-trajectory replay | 40% | 100% | 46% | 50% | ||||
| Imitation-trained ILA policies | ||||||||
| FM-ILA | 75% | 95% | 50% | 100% | ||||
| VLA-ILA | 50% | 85% | 37% | 95% | ||||
| WAM-ILA | 70% | 80% | 45% | 100% | ||||
Table 7: ILA policies and action-trajectory replay. We report SR and L1 (mean standard deviation).
Figure 9: Successful-window reweighting across four tasks. Left: empirical weight distributions. Right: mean weight within each reward decile. Dashed lines indicate uniform weighting, . Weights are recomputed from cached rewards and the fixed calibration of each completed run, using , , and . The diagnostic pools contain successful-rollout policy predictions.
Figure 10: Observation-compatible pairs across four tasks. Failure and reference data occupy the upper and lower rows of each task. Wrist images and tactile maps are taken at the respective window starts. The four tactile maps show local tangential deformation magnitudes, with a shared color scale within each task. The tactile curve sums the four per-sensor spatial XY resultant magnitudes in raw deformation units. TCP shows the mean positional displacement of the two recorded end effectors from their respective window starts.
Figure 11: Four tasks encoded by a shared Stage-1 model. Each point represents a 128-dimensional token and is colored by task. All tasks are encoded by the same model with shared preprocessing.
| Task | Model | Action | Tactile | Force |
|---|---|---|---|---|
| Cucumber peeling | Task-specific | 34.21 | 3.50 | 25.80 |
| Shared | 22.63 | 3.51 | 23.72 | |
| Egg flipping | Task-specific | 18.05 | 2.39 | 25.37 |
| Shared | 6.12 | 2.87 | 16.54 | |
| Celery cutting | Task-specific | 6.90 | 1.21 | 14.25 |
Table 8: Shared versus task-specific reconstruction. Action and tactile report L1 errors in the original recorded coordinates. Force reports magnitude L1 of the directly decoded aggregate deformation, a force proxy in sensor-recording units rather than newtons.
Figure 12: Successful task interactions across four tasks. The sequences show strip formation and transfer, egg turnover, plate wiping, and celery separation and transfer. The peeling row depicts one completed peel–place cycle, rather than all three cycles required for rollout success. Cutting is assessed per cut; the displayed separated pieces and transfer do not imply that every preceding cut succeeded.
| Task | Required motion–contact coordination | Cases |
|---|---|---|
| Peeling | Maintain a productive stroke; acquire and transfer the strip. | P1–P4 |
| Flipping | Support, turn, and release the egg while retaining the tool. | E1–E5 |
| Wiping | Retain both objects and move effective contact across the marks. | W1–W3 |
| Cutting | Stabilize, sever, and clear the knife between strokes. | C1–C5 |
Table 9: Interaction requirements illustrated by the rollout cases. Case IDs correspond to the rows of Figs. 13 – 15 .
Figure 13: Failure cases in cucumber peeling and plate wiping. P1–P4 show missed acquisition, strip breakage, premature lifting, and tool overrun. W1–W3 show an unstable sponge grasp, residual staining despite a retained grasp, and prolonged wiping with limited coverage. W2–W3 end with enlarged views of residual markings. The images do not determine contact pressure.
Figure 14: Failure cases in egg flipping. The rows distinguish attachment during rotation (E1), missing or inadequate support (E2–E3), withdrawal without turnover (E4), and premature tool release (E5). Frames retain the observed tool and object states; they do not isolate insertion depth or lifting clearance as a unique cause of failure.
Figure 15: Failure cases in celery cutting. C1–C5 illustrate incomplete separation, inadequate blade clearance, loss of stabilization, and unstable acquisition. A cut succeeds only if the celery is completely severed at the cutting location before the knife clears the stalk. Residual connections, including cases where the partially cut stalk follows the lifted knife, count as failures. Rows show selected failure phases; individual cuts within the same recording may have different outcomes.