What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory

Amir-Hossein Shahidzadeh; Seungjae Lee; Eadom Dessalene; Shanthosh Raaj Mohanram Mageswari; Soroush Etemad; Furong Huang; Cornelia Fermüller; Yiannis Aloimonos

cs.ROSep 29, 2026

What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory

Authors: Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene, Shanthosh Raaj Mohanram Mageswari, Soroush Etemad, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos

Organizations: Computer Science Department, University of Maryland, College Park, MD, USA

Abstract

Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a skill-conditioned representation in which the queried skill conditions fusion over modality-specific short-term observation tokens while attending to a sparse event memory that retains terminal observations from the last $K$ executed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation not only policy architecture is a central challenge in multi-modal representation. Project Website: http://what-to-attend-what-to-keep.github.io/

Figures & tables

Figure 1: SkillFormer learns what to attend to and what to remember. Skill-conditioned fusion combines visual and tactile observation windows with progress-guided event memory. Tactile observations reveal tightening and slip events that are difficult to distinguish visually, while memory helps distinguish repeated inspections across boxes. By connecting event detection with retention, S kill F ormer turns brief sensory observations into lasting context for interpreting subsequent interactions.

Figure 2: Pearson

r

measures correlation between predicted and ground-truth progress (higher is better). Greater variability on twisting w/o event memory suggests less consistent performance that does not establish online generalization. ∗ : fine-tuned; eye/fingerprint: vision/touch.

Figure 3: S kill F ormer on blind cube search , queried with “inspect box 2” . What to attend: the two tactile streams are encoded frame-by-frame by FeelAnyForce and the RGB stream by CLIP ViT (both fine-tuned), each over its own horizon; the skill description is encoded by the frozen CLIP text tower. Every token is the sum of its feature and a temporal, modality, and event embedding, so the transformer knows when a token was observed, from which sensor, and whether it belongs to the current window or to memory. What to keep: when a skill completes (here inspect box 1 , withdraw box 1 , Reach box 2 ) their encoded observation window is stored as event tokens and attended alongside the current observation. The learned output tokens are mapped to predict queried skill progress.

Figure 4: UMI-Tac. a handheld gripper with a wrist camera and tactile sensors.

Method	Metric	Lamp replacement		Bottle capping
Method	Metric	Reach bulb	Twist bulb	Reach cap	Slide cap
Robometer Liang et al. (2026)	Pearson $r$ $\uparrow$	$0.638\pm 0.054$	$0.209\pm 0.053$	$0.819\pm 0.084$	$0.578\pm 0.058$
Robometer Liang et al. (2026)	MSE ( $\times 10^{-3}$ ) $\downarrow$	$84.5\pm 19.6$	$499.8\pm 59.3$	$46.8\pm 17.4$	$232.2\pm 42.2$
Robometer ∗ Liang et al. (2026)	Pearson $r$ $\uparrow$	$0.886\pm 0.032$	$0.370\pm 0.106$	$0.885\pm 0.018$	$0.875\pm 0.020$
Robometer ∗ Liang et al. (2026)	MSE ( $\times 10^{-3}$ ) $\downarrow$	$27.4\pm 6.2$	$98.6\pm 46.3$	$51.3\pm 9.5$	$40.7\pm 4.3$
ReWiND ∗ Zhang et al. (2025)	Pearson $r$ $\uparrow$	$0.847\pm 0.060$	$0.635\pm 0.136$	$0.880\pm 0.035$	$0.878\pm 0.037$

Table 1: Progress estimation results across various visual or tactile dominant skills.( bold and underline are based on the best mean)

Panel of Figure 5 — Figure 5: We show observations and their timesteps which correspond to moments when an event is being detected where progress crosses the event threshold for a sample discrete (top) and conditional skill (bottom) in Lamp Replacement and Blind Cube Search tasks.

Panel of Figure 6 — Figure 6: t-SNE projections of produced transformer tokens colored by ground-truth progress. Inner/outer image borders indicate gt/predicted progress.

Method	Delay@0.9 $\downarrow$	MSE ( $\times 10^{-3}$ ) $\downarrow$	Pearson $r$ $\uparrow$
w/o Events	$63.46\pm 119.43$	$143.6\pm 220.0$	$0.683\pm 0.301$
FiLM	$77.81\pm 118.55$	$116.0\pm 253.4$	$0.727\pm 0.263$
S kill F ormer (K=1)	$42.77\pm 79.83$	$83.1\pm 201.7$	$0.791\pm 0.287$
Vision Only (K=3)	$37.46\pm 87.46$	$57.1\pm 155.1$	$0.857\pm 0.213$
S kill F ormer (K=3)	$16.10\pm 37.20$	$11.6\pm 34.1$	$0.953\pm 0.082$

Table 2: Blind search results averaged over all subtasks.

Method	Metric	Lamp replacement				Bottle capping
Method	Metric	Reach bulb	Align bulb	Twist bulb	Overall	Reach cap	Align cap	Slide cap	Overall
Vision Only	Delay@0.9	$3.75\pm 2.09$	$15.50\pm 17.22$	$19.75\pm 8.69$	$10.08\pm 11.48$	$5.36\pm 3.31$	$29.64\pm 25.97$	$82.18\pm 82.08$	$27.67\pm 46.81$
	MSE ( $\times 10^{-3}$ )	$0.2\pm 0.1$	$13.8\pm 18.8$	$9.3\pm 6.4$	$5.4\pm 9.7$	$0.3\pm 0.3$	$18.4\pm 14.8$	$72.7\pm 45.5$	$17.8\pm 32.7$
	Pearson $r$	$0.997\pm 0.002$	$0.967\pm 0.039$	$0.842\pm 0.095$	$0.904\pm 0.160$	$0.997\pm 0.003$	$0.963\pm 0.028$	$0.798\pm 0.103$	$0.950\pm 0.085$
w/o Events	Delay@0.9	$4.17\pm 6.24$	$15.50\pm 15.96$	$9.92\pm 9.30$	$7.92\pm 10.41$	$5.00\pm 3.41$	$17.18\pm 8.63$	$41.09\pm 14.99$	$17.36\pm 15.50$
	MSE ( $\times 10^{-3}$ )	$0.4\pm 0.8$	$40.3\pm 53.0$	$8.4\pm 11.6$	$9.0\pm 26.4$	$0.3\pm 0.2$	$8.1\pm 7.7$	$46.8\pm 13.1$	$11.2\pm 18.1$

Table 3: Progress estimation results across various tasks and skills. Overall metrics are reported over the entire skills in each task.

Figure 7: Robustness to visual, geometric, temporal, and modality perturbations across three tasks. closer to the center is better. Colored letters identify the subtask with the largest error increase over clean inputs (R: reach, A: align, S: slide, T: twist, I: inspect, W: withdraw).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Tactile evolution during normal and final bulb twisting. During the last twist (top row), the tactile image shows an increasing-force pattern as resistance builds near the end of the thread. During normal twisting, the contact pattern remains relatively stable and changes only slightly across frames. This temporal tactile cue defines completion of the twist bulb skill.

Method	Metric	Reach box 1	Inspect box 1	Inspect box 2	Withdraw 2	Overall
w/o Events	Delay@0.9 $\downarrow$	$25.57\pm 12.53$	$31.00\pm 15.43$	$36.10\pm 11.55$	$66.20\pm 96.58$	$63.46\pm 119.43$
	MSE ( $\times 10^{-3}$ ) $\downarrow$	$23.5\pm 19.6$	$496.0\pm 455.7$	$287.1\pm 255.4$	$131.4\pm 89.4$	$143.6\pm 220.0$
	Pearson $r$ $\uparrow$	$0.833\pm 0.128$	$0.355\pm 0.350$	$0.462\pm 0.449$	$0.580\pm 0.116$	$0.683\pm 0.301$
FiLM	Delay@0.9 $\downarrow$	$58.14\pm 25.27$	$47.50\pm 18.94$	$100.80\pm 46.24$	$63.00\pm 92.80$	$77.81\pm 118.55$
	MSE ( $\times 10^{-3}$ ) $\downarrow$	$14.7\pm 13.2$	$495.1\pm 649.8$	$171.7\pm 132.2$	$56.0\pm 74.4$	$116.0\pm 253.4$
	Pearson $r$ $\uparrow$	$0.868\pm 0.101$	$0.400\pm 0.316$	$0.621\pm 0.238$	$0.575\pm 0.250$	$0.727\pm 0.263$

Appendix

Table 4: Progress estimation results across blind cube search skills.

Panel of Table 5 — Table 5: Skill definitions used for progress supervision.

Figure 9: Contact-rich manipulation tasks and their representation requirements. Left: Lamp replacement involves grasping, socket alignment, and twisting until tightened. Middle: Bottle capping involves grasping, thread alignment, and sliding, with slip providing a salient tactile event. Right: Blind cube search involves successive inspections and withdrawal after an unsuccessful attempt, followed by insertion into a puzzle box once the target is found. Arrows indicate primitive skills. Large images highlight a task execution; orange-bordered insets show egocentric RGB views, and red-bordered insets show tactile readings from the two fingers. Together, the tasks illustrate skill-specific tactile patterns and complementary needs for sensory history and memory of prior interactions for estimating their progress.

Explore similar work

Jun 4, 2026cs.RO

Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation

Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose a novel architecture using modality-specific convolutional stems and transformer-based fusion that effectively fuses information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multi-sensor representation then conditions a flow-matching policy for solving downstream tasks. Experimental results across five contact-rich manipulation tasks demonstrate the effectiveness of multi-resolution tactile features in imitation learning. MiTaS achieves an average success rate of 80 %, while vision-only (31 %) and visual-tactile (54 %) baselines cannot solve the task reliably. Co-training a visuo-tactile model with multi-tactile data boosts performance by over 10 % in certain tasks, without having access to the Evetac sensor during policy evaluation. A detailed sensor-reading and attention analysis reveals the importance of different sensors throughout task execution, validating our multi-resolution tactile sensing approach. Project Page: http://mitas-touch.github.io.

Rickmer Krohn, Erik Helmut, Niklas Funk +3

Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1

Tactile Contact-Rich Dexterous Manipulation Imitation Learning

Ask PDF

Aug 21, 2026cs.RO

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.

Congsheng Xu, Qiaochu Yang, Fangyuan Shi +9

Shanghai Jiao Tong University · Xense Robotics

Tactile Contact-Rich Dexterous Manipulation

Ask PDF

Jan 31, 2026cs.RO

Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation

Tactile feedback is important for contact-rich robotic manipulation, yet effective use of tactile observations remains challenging when tactile signals are image-like, hardware-dependent, and only weakly aligned with external visual observations. This study addresses this representation-learning problem by proposing a visuo-tactile contrastive learning framework for imitation-based manipulation. The method aligns external RGB observations and calibrated tactile images in a shared embedding space using a CLIP-style objective, and integrates the resulting representation into an Action Chunking Transformer (ACT) policy. A low-cost visuo-tactile gripper (LVTG) is proposed to provide a modular and durable sensing platform for reproducible data collection, supplying tactile observations that can be used by downstream manipulation algorithms. Experiments on contact-rich manipulation tasks show that tactile feedback improves the average task completion rate from 30% for a vision-only ACT baseline to 42%, and that the proposed contrastive pretraining further increases the completion rate to 54%. These results indicate that explicitly aligning visual and tactile observations provides more useful contact-aware features for downstream policy learning than directly adding tactile images without pretraining.

Yaohua Liu, Rong Fu, Amir H. Gandomi +2

Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3

Tactile Contact-Rich Dexterous Manipulation Contact-Rich Manipulation Haptic Feedback Gripper Contrastive Learning

Ask PDF