Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore · Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore
Integrating force awareness into Vision-Language-Action (VLA) models involves not merely encoding wrench measurements as an additional modality, but determining when and how strongly physical interaction should influence robot action generation. Existing VLAs that use force typically inject it through a fixed pathway, without explicitly determining when it is relevant to the current action. This calls for an intermediate force relevance mechanism inside the VLA, in which visual-language context estimates the relevance of physical feedback to the current task before force enters the action pathway. FR-VLA addresses this gap with a Vision-Language Force-Reasoning Model that uses learnable force query tokens to derive a force admission signal grounded in visual-language context, and uses this signal to modulate an external tokenized force pathway for action generation. In real-world experiments on five contact-rich tasks across two robotic platforms, FR-VLA achieves the highest mean success rate over six evaluated stages, and trajectory visualizations show that its force admission varies with the interaction phase.
Figures & tables
Figure 1: (a) Two representative force-aware VLA designs either inject force into the VLM token stream or condition the action expert directly. (b) A Switch Cable Connection trajectory shows synchronized camera observations, six-dimensional wrench measurements, and the force admission signal across task stages.
Figure 2: Overview of FR-VLA. (a) Overall architecture: four token groups are processed by the VLM-based Implicit Force Reasoning Module, which infers contact-relevant motion phases and produces a force admission signal. Measured wrench histories are encoded by the Force Tokenizer Module and enter the Action Expert only through the admission-conditioned action-side pathway. (b) The VLM-based Implicit Force Reasoning Module grounds force query states with visual features through cross-attention and decodes them into motion phase inference, producing the admission signal that determines when force information becomes action-relevant. (c) The Force Tokenizer Module discretizes wrench histories into contact-semantic codes via a frozen VQ-Encoder and uses their Code Embeddings to FiLM-condition the force pathway.
Figure 3: Trajectory-level comparison between raw wrench dynamics and learned force admission. Top: raw 6-DoF wrench during connector insertion. Bottom: force admission signal in FR-VLA gtf (solid) sustains force eligibility across contact-relevant windows, whereas a direct injection baseline (dashed) tracks wrench magnitude and spikes only at large transients.
Figure 4: Real-world contact-rich manipulation benchmark. The evaluation suite spans Gear Assembly, Switch Cable Connection with a visually occluded side power connector stage, Curved Surface Cleaning, Screw Tightening, and Seat Belt Buckle Insertion, covering contact transitions, constrained insertion, sustained surface interaction, and visually limited contact reasoning.
Figure 5: Real-world success rates. Per-task/stage success rate over 20 trials for each method; FR-VLA (ours) is shown in purple. Switch Cable Connection is evaluated in two stages, covering top Ethernet cable insertion and side power connector insertion. VLA-Adapter w/F denotes directly inputting time-aligned 6D force together with robot state.
Figure 6: Force admission trajectory in Switch Cable Connection. Keyframes, end-effector height, 6D wrench, and admission signal gtf over a rollout with two retractions before insertion is completed. Shaded regions denote inferred phases; admission stays positive throughout the inferred contact phase and becomes negative once the phase switches to free.
Task group
Task
Balanced acc.
Insertion
Gear Assembly
93.7%
Insertion
Switch Cable Connection
90.5%
Insertion
Seat Belt Buckle Insertion
88.5%
Sustained contact
Curved Surface Cleaning
80.6%
Sustained contact
Screw Tightening
86.2%
Table 1: Phase inference accuracy. Balanced accuracy after merging approach into free. Switch Cable Connection averages its two stages.
Variant
Gear
Switch-Top
Switch-Side
Cleaning
Screw
Buckle
Avg.
w/o force admission
60%
30%
15%
50%
25%
30%
35.0%
w/o force tokenization
70%
40%
20%
55%
20%
35%
40.0%
FR-VLA
90%
60%
35%
70%
40%
55%
58.3%
Table 2: Ablation results on force admission and force tokenization.
Method
Unseen gear
Target perturb.
VLA-Adapter w/F
60%
50%
FR-VLA
85%
75%
Table 3: Gear Assembly robustness over 20 trials per perturbation category.
Group 1
Group 2
Group 3
Group 4
4/10
6/10
5/10
5/10
Table 4: Switch Cable Connection under unseen initial cable positions, with 10 trials per group.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Robot platforms and camera setup. The two evaluation platforms, UR5e (left) and UR3e (right), each use the robot’s built-in 6D force/torque sensing. Both share the same camera configuration: a third-view Intel RealSense D455 camera and a wrist-mounted Intel RealSense D405 camera, highlighted on each platform.
Figure 8: Wrench magnitude gating on Gear Assembly. Six-dimensional wrench measurements and the admission signal of the magnitude gating baseline during a representative execution. The admission curve varies with the measured wrench amplitude.
Figure 9: Phase inference, force codes, and force admission across tasks. Frame borders and shaded regions denote predicted phases, and the code strip shows the selected VQ code kt . Additional trajectories are shown in Figure 10 .
Figure 10: Additional phase inference, force codes, and force admission trajectories. Continuation of Figure 9 ; frame borders, shaded regions, and the code strip use the same conventions.
Task
Manipulation segments
Training phases
Gear Assembly
1
Free, approach, contact
Switch Cable Connection
2
Free, approach, contact
Curved Surface Cleaning
1
Free, contact
Screw Tightening
1
Free, contact
Seat Belt Buckle Insertion
1
Free, contact
Appendix
Table 5: Phase supervision configuration. Contact detection uses shared thresholds; the training phase vocabulary differs by task.
Task
Free
Contact
Gear Assembly
64%
36%
Switch Cable Connection
53%
47%
Curved Surface Cleaning
44%
56%
Screw Tightening
56%
44%
Seat Belt Buckle Insertion
63%
37%
Appendix
Table 6: Distribution of offline motion phase labels. Relative prevalence of training labels after merging approach into free. Free includes approach; contact is unchanged. Switch Cable Connection is reported as one task rather than separating its two execution stages.
Figure 11: Association between force codes and interaction events. Each cell shows the lift P(event∣code)/P(event) on a logarithmic color scale; red indicates enrichment and blue indicates depletion.
Figure 12: Additional robustness test settings. (a) Target peg position changes. (b) Unseen gears. (c) Initial Ethernet cable position changes.
Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries. Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/
Ruicheng Li, Qixiu Li, Ruichun Ma +8
Tsinghua University · Microsoft Research · Fudan University +1
Humanoid robots are expected to perform diverse human-level tasks in daily environments, many of which require precise regulation of interaction forces. While recent vision-language-action (VLA) models have shown promise for semantic planning and visuomotor control, existing humanoid systems primarily represent actions through geometric motion goals and rely on whole-body controllers focused on motion tracking, with limited explicit reasoning or control of interaction forces. This limitation is particularly relevant in contact-rich tasks, where geometrically similar motions may require different force regimes depending on the task context and where visual observations may become unreliable after contact. In this work, we present Opt2VLA, a force-aware VLA framework that introduces explicit force commands at the VLA-to-control interface for humanoid whole-body manipulation. A single multi-task VLA policy jointly predicts both geometric motion goals and continuous contact-force references, which are tracked by task-specific reinforcement learning (RL)-based whole-body controllers. To provide scalable and physically grounded supervision, we generate dynamically feasible and contact-consistent training data via whole-body trajectory optimization (TO) with explicit force references. We evaluate Opt2VLA on three contact-rich humanoid tasks and show that explicit force conditioning enables more accurate and consistent force regulation than motion-only control, while physically grounded torque supervision from TO further improves force tracking accuracy and stability. Closed-loop evaluations further demonstrate language-conditioned force modulation with Opt2VLA in simulation and on humanoid hardware.
Fukang Liu, Yipu Chen, Jaehwi Jang +3
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA, USA
Force/torque feedback can substantially improve Vision-Language-Action (VLA) models on contact-rich manipulation, but most existing approaches fuse all modalities at a single operating frequency. This design ignores the mismatched sampling rates of real robot sensors, forcing downsampling of the high-frequency contact cues needed for reactive correction. Combined with common VLM-action-expert (AE) pipelines that execute action chunks largely open loop between expensive VLM updates, unified-frequency fusion often yields delayed responses to impacts, stick-slip, and force spikes. We propose FAVLA, a force-adaptive fast-slow VLA that decouples slow perception planning from fast contact-aware control. FAVLA runs a slow VLM at a fixed low frequency to encode modalities to produce latent representations and to predict near-future force variation. A fast AE then executes at a variable high frequency, conditioning on the latest force sequence data to generate reactive actions. We further introduce a force adapter that injects high-frequency force features into multiple AE layers, and adaptively schedules the AE's execution frequency based on the VLM's predicted force variation. Extensive experiments on contact-rich tasks demonstrate that FAVLA significantly outperforms baselines, achieving superior reactivity and success rates, especially with a smaller contact force during manipulation.
Yao Li, Peiyuan Tang, Wuyang Zhang +7
University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China · Central South University, Changsha, China