In visually ambiguous manipulation such as detecting button click tactile feedback is often the sole source of ground truth. However, fusing tactile data poses a significant challenge due to a spatiotemporal mismatch: tactile perception requires high-frequency processing with long-horizon memory (System 1), whereas visual policies operate at low control frequencies (System 2). Existing architectures struggle to bridge this gap: Transformers are computationally prohibitive for high-frequency loops (>100Hz), while LSTMs suffer from forgetting over extended interaction histories. In this paper, we introduce TacMamba, a hierarchical architecture that aligns high-bandwidth tactile reflexes with low-frequency visual planning. Our approach comprises three core contributions: (1) a custom high-frequency tactile interface designed for flexible integration; (2) a Mamba-based Tactile History Compressor that encodes continuous force history into a compact state with O(1) inference latency (0.45 ms), enabling plug-and-play fusion with VLA models without joint pre-training and (3) a Tactile-Guided Dual-Stage Training strategy that leverages temporal discrimination for self-supervised representation learning and phase-uniform sampling to mitigate data sparsity. Experiments on discrete counting and implicit state switching demonstrate that TacMamba achieves 100% success rates, significantly outperforming the visual-only pi_0.5 baseline, while strictly satisfying hard real-time constraints.
Figures & tables
Fig. 1: The TacMamba System Architecture. The framework bridges the spatiotemporal discrepancy between high-speed reflexes and low-frequency reasoning. Left (System 1): The tactile encoder processes 1D force streams at 100Hz using Mamba Models, recursively updating the hidden state ht in real-time. Right (System 2): This compressed hidden state ht is projected and asynchronously injected as a soft prompt into the low-frequency ( ∼ 1Hz) Vision-Language-Action (VLA) planner.
Fig. 2: System Overview. (a) Modular morphology-based tactile fingertip design, where an integrated compliant contact body supports both fingertip and fingerpad interactions and mechanically projects distributed contacts onto a single-axis force sensor; lateral side panels protect the internal structure, and a reconfigurable dual-clamp interface enables attachment to generic parallel grippers. (b) FEM-based characterization of force projection, showing peak contact pressure versus applied load for fingertip and fingerpad contacts, together with (c) representative visualizations of total deformation, equivalent (von Mises) stress, and contact pressure.
Fig. 3: The TacMamba Network Architecture. Top: The core TacMamba backbone processes continuous tactile streams via a hierarchical Selective SSM, utilizing RevIN and Channel Independence for feature extraction. Bottom-Left: An expanded view of the Mamba block mechanism, illustrating how input-dependent parameters ( Δt,Bt,Ct ) model hybrid dynamics. Bottom-Right: The auxiliary discriminator network, employed exclusively during the training phase to facilitate robust feature learning and enforce temporal causality.
Fig. 4: Efficiency Analysis. Inference latency and memory growth.
Figure 5
Fig. 7: Qualitative folding results comparison. (a) Successful cloth folding workflow executed by the full TacMamba model. The zoomed-in views highlight precise and stable grasps on the fabric corners enabled by high-frequency tactile feedback. (b) and (c) Typical failure modes observed when the phase-prioritized sampling strategy is disabled (ablation): ”Failed to grasp” (missing the corner due to visual ambiguity) and ”Shape crumpled” (improper tension control causing deformation).
Method
Success Rate
External Visual Baselines ∗ :
OpenVLA [ 29 ]
6%
CogACT [ 30 ]
15%
MemoryVLA [ 11 ]
58%
Our Implementations:
π0.5 [ 2 ] (Early stage, 5k steps)
37.5%
TABLE II: Performance on Sequential Button Pressing Task
Tactile perception is indispensable for contact-rich manipulation, yet integrating it into Vision-Language-Action (VLA) models often induces modality collapse, where high-bandwidth visual features overshadow sparse tactile cues. Inspired by Predictive Coding, a neural mechanism where the brain attenuates predictable inputs to prioritize surprising stimuli, we propose ResTacVLA. Rather than treating tactile data as raw input, we reformulate it as a Residual Tactile Representation capturing the discrepancy between visual priors and physical sensations. By filtering out visually predictable dynamics, this formulation transforms sparse tactile signals into dense, high-value information gain, thereby inherently resolving the bandwidth mismatch. These residuals are discretized through a Vector Quantized (VQ) bottleneck into Latent Contact Primitives that capture critical events missed by vision. Analogous to the neural surprise signal, we leverage the uncertainty of the visual prior to adaptively gate tactile integration, prioritizing residuals specifically during visually unreliable phases to explicitly prevent visual dominance. Experimental results show that ResTacVLA consistently outperforms all baselines on a diverse set of contact-rich manipulation tasks, while remaining robust to unexpected dynamic disturbances. Project page: https://awilekong.github.io/ResTacVLA/
Pengwei Zhang, Bin Xie, Ce Hao +5
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Zhongguancun Academy, Beijing 100094, China · Dexmal, Beijing 100096, China
The ability to react dynamically to tactile signals has long been considered crucial to agile human-level dexterity. Yet contemporary learning-based Vision-Language-Action (VLA) models for robotic manipulation generally either overlook the tactile modality or are limited to encoders with static cues, due in part to the scarcity of diverse training data and standardized evaluation, architectural constraints in current VLA models, and limitations of static tactile encoders. In this paper, we push the frontier of tactile-reactive manipulation by addressing all of these limitations. We propose a large-scale, 100-hour tactile-rich dataset collected via a novel, data-efficient recipe that prioritizes elementary motor primitives. To effectively exploit naturally high-frequency touch signals without sacrificing the existing capabilities of existing VLAs, we introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder. We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation, achieving over 30% higher average success rate than the strongest baseline.
Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.
Shuai Tian, Yupeng Zheng, Yuhang Zheng +7
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · TARS Robotics +2