cs.ROOct 3, 2026

AgenticTactileVLA: Contact-Guided Execution-Time Supervision for Generalizable Dexterous Manipulation without VLA Retraining

Authors: Elizaveta Semenyakina, Ivan Snegirev, Mikhail Kiselev, Miguel Altamirano Cabrera, Artem Lykov, Hajira Amjad, Dzmitry Tsetserukou

Organizations: Intelligent Space Robotics Lab, Skolkovo Institute of Science and Technology, Moscow, Russian Federation. · R&D Center, MWS, Moscow, Russian Federation.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language-action policies may predict a transferable manipulation strategy yet fail to realize it reliably on the encountered object: objects compatible with the same grasp differ in geometry and compliance, and visual feedback degrades under closure occlusion. AgenticTactileVLA is presented as an execution-time supervisor that shifts part of object-specific adaptation from prediction to physical interaction. A fixed VLA provides the approach and hand targets; the supervisor decides whether to remain transparent, refine finger flexion, retain or release the corrected configuration, return control to the VLA for retry, or select a compliant hand-control regime. It uses finger-position and motor-effort feedback as proprioceptive contact evidence and requires neither tactile sensors nor VLA retraining. On a Unitree G1 with a BrainCo Revo2 hand, a randomized matched-block evaluation on five objects held out from VLA training yields 61.3% completion for the base VLA, 72.0% for unconditional close-to-stall control, and 84.0% for the supervisor under a shared budget; the gain is positive on every object and persists under moderate pose perturbations. Ablations show the gain is not explained by extended closure alone, and that selective triggering reduces correction episodes by 65.7% with no detected change in completion. A retention audit shows acceptance predicts retention in 88.9% of held-out cases, while compliant objects expose conservative false rejection. A thin-walled-cup study demonstrates contextual routing to compliant control, matching an always-compliant reference. These results suggest that contact-guided execution-time adaptation can improve the object-level generalization of a fixed VLA to held-out objects by adapting physical realization without object-specific retraining.

Figures & tables

Fig. 1: From VLA prediction to execution-time physical adaptation. The fixed VLA provides the approach and initial hand configuration, while the execution agent adapts finger closure to the encountered object.
Fig. 2: System architecture of AgenticTactileVLA. The fixed VLA provides the nominal manipulation strategy, while the execution supervisor monitors its physical realization and intervenes only when adaptation is required.
Statebtb_{t}OverrideTransition / exit condition
Idle0noupcoming closure in the action chunk
Armed0noterminal grasp assessment
Adjust1yesgrasp verdict or timeout
Hold0yespolicy reopens the hand
TABLE I: Execution-agent states. btb_{t} gates body-reference progression; the override column indicates whether the agent replaces the nominal hand target.
ObjectVLAVLA + stallSupervisor
Cube9/1511/1513/15
Fish9/1511/1513/15
Soy sauce bottle11/1512/1514/15
Thermos lid7/158/1510/15
Hand cream tube10/1512/1513/15
Pooled46/7554/7563/75
TABLE II: Fixed-budget task completion under randomized, interleaved conditions. All conditions use 15 trials per held-out object; intervals are 95% Wilson.
VariantCompletionRateCorrections
U-stall72/10072%135
U-full84/10084%134
S-position79/10079%49
S-retain77/10077%47
S-full86/10086%46
TABLE III: Component-controlled ablation over 100 matched blocks. Corrections count intervention episodes.
RegimeVLAVLA + stallSupervisor
Nominal32/5037/5043/50
Translation25/5032/5039/50
Yaw23/5030/5037/50
Pooled80/15099/150119/150
TABLE IV: Task completion under initial-pose perturbations.
Fig. 3: Diagnostic contact detection (a) and detector ineligibility due to insufficient remaining finger travel (b).
Support predicateRetainedNot retainedTotal
At least three70676
Below three182644
Total8832120
TABLE V: Support verdict versus object retention during the diagnostic lift.
Mode selected byTrialsIntactDamagedCompliant active
No selection5050/5
Instruction cue5505/5
VLM from image5414/5
Instruction + VLM5505/5
Always compliant5505/5
TABLE VI: Thin-walled-cup integrity and compliant-mode activation.
Fig. 4: Representative thin-walled-cup outcomes. (a) Damage under nominal position control. (b) Preserved integrity under compliant control.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,704 papers

Explore similar work

CardsList
Aug 16, 2026

ViTaR: Visuo-Tactile Residual Adaptation for Foundation VLA Manipulation

As Vision-Language-Action (VLA) models scale toward real-world deployment, contact-rich manipulation exposes a critical blind spot: these policies encode broad visual-semantic priors yet remain unaware of local contact events, producing identical actions whether contact is established, lost, or destabilized. Existing remedies either modify VLA internals, risking catastrophic forgetting, or demand online reinforcement under near-failure contact conditions. Both grant tactile unbounded influence over action generation, conflicting with the priors that make VLAs generalizable. We introduce ViTaR, which reframes tactile feedback from an action-generating perceptual input to an execution modulator that selects and scales bounded residual corrections atop a frozen VLA, preserving pretrained capabilities by construction. ViTaR decomposes adaptation into two stages: Effect-Guided Modeling determines whether and which correction is locally justified via outcome-grounded preference evidence, and Residual Action Modulation converts this evidence into a residual choice with continuously scaled gain from real-time visuotactile observations. On the UniVTAC benchmark spanning seven contact-rich tasks, ViTaR achieves 61.3% average success, a 30.6 percentage-point improvement over its frozen VLA base that also surpasses purpose-built tactile baselines. Physical-robot experiments confirm that bounded tactile modulation transfers to real sensor noise and dynamics.

Yi Wang, Renjun Wu, Jinyan Liu +1

Beijing Institute of Technology

Contact-Rich Robotic ManipulationTactile-Vision-Language-Action ModelsVision-Language-Action ModelsTactile Sensing for Robotic ManipulationResidual Policy Learning
AskPDF
Jul 9, 2026

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://github.com/RLinf/RPent.

Yixian Zhang, Huanming Zhang, Feng Gao +14

Tsinghua University · Striding AI · Purdue University +4

Long-Horizon Robotic ManipulationLanguage-Conditioned Robot ManipulationGeneralization in Robotic ManipulationVision-Language-Action ModelsTest-Time Adaptation of VLA ModelsRobustness of VLA Models+1
AskPDF
May 8, 2026

AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Models

Vision-Language-Action (VLA) models have significantly advanced the capabilities of robotic agents in executing diverse tasks; however, they still face challenges in contact-rich manipulation scenarios that require precise physical interactions. To address this limitation, recent studies have attempted to incorporate tactile signals during downstream tasks, enabling pretrained VLAs to interpret tactile feedback. Nevertheless, introducing new modalities during finetuning, which are rarely present in the pretrain stage, may disrupt the pretrained capabilities of VLAs. In addition, the inherently slow inference speed of VLAs hampers real-time responsiveness and limits the effective utilization of tactile feedback for action adjustment. To overcome these challenges, we propose Adaptive Tactile Vision-Language-Action (AT-VLA), which introduces a novel Adaptive Tactile Injection mechanism. This mechanism dynamically determines the appropriate timing and locations for tactile injection, incorporating only when it significantly contributes to action generation, thereby minimizing interference with pretrained representations. Furthermore, to enable rapid and accurate tactile responses, we propose a Tactile Reaction Dual-Stream mechanism, which decouples sensory processing into a slow visual-language stream for low-frequency perceptual reasoning and a fast tactile control stream for high-frequency physical interaction understanding, achieving real-time close-loop responses within 0.04 s. Real-world experiments thoroughly validate the effectiveness of AT-VLA in contact-rich manipulation tasks. The project page is available at: https://sites.google.com/view/at-vla.

Xiaoqi Li, Muhe Cai, Jiadong Xu +5

School of Computer Science, Peking University · PrimeBot · PKU Lab

Contact-Rich Robotic ManipulationVLM AdaptationHaptic FeedbackVision-Language-Action ModelsTactile Sensing for Robotic Manipulation
AskPDF