Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore · Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore
Integrating force awareness into Vision-Language-Action (VLA) models involves not merely encoding wrench measurements as an additional modality, but determining when and how strongly physical interaction should influence robot action generation. Existing VLAs that use force typically inject it through a fixed pathway, without explicitly determining when it is relevant to the current action. This calls for an intermediate force relevance mechanism inside the VLA, in which visual-language context estimates the relevance of physical feedback to the current task before force enters the action pathway. FR-VLA addresses this gap with a Vision-Language Force-Reasoning Model that uses learnable force query tokens to derive a force admission signal grounded in visual-language context, and uses this signal to modulate an external tokenized force pathway for action generation. In real-world experiments on five contact-rich tasks across two robotic platforms, FR-VLA achieves the highest mean success rate over six evaluated stages, and trajectory visualizations show that its force admission varies with the interaction phase.
Figures & tables
Figure 1: (a) Two representative force-aware VLA designs either inject force into the VLM token stream or condition the action expert directly. (b) A Switch Cable Connection trajectory shows synchronized camera observations, six-dimensional wrench measurements, and the force admission signal across task stages.
Figure 2: Overview of FR-VLA. (a) Overall architecture: four token groups are processed by the VLM-based Implicit Force Reasoning Module, which infers contact-relevant motion phases and produces a force admission signal. Measured wrench histories are encoded by the Force Tokenizer Module and enter the Action Expert only through the admission-conditioned action-side pathway. (b) The VLM-based Implicit Force Reasoning Module grounds force query states with visual features through cross-attention and decodes them into motion phase inference, producing the admission signal that determines when force information becomes action-relevant. (c) The Force Tokenizer Module discretizes wrench histories into contact-semantic codes via a frozen VQ-Encoder and uses their Code Embeddings to FiLM-condition the force pathway.
Figure 3: Trajectory-level comparison between raw wrench dynamics and learned force admission. Top: raw 6-DoF wrench during connector insertion. Bottom: force admission signal in FR-VLA gtf (solid) sustains force eligibility across contact-relevant windows, whereas a direct injection baseline (dashed) tracks wrench magnitude and spikes only at large transients.
Figure 4: Real-world contact-rich manipulation benchmark. The evaluation suite spans Gear Assembly, Switch Cable Connection with a visually occluded side power connector stage, Curved Surface Cleaning, Screw Tightening, and Seat Belt Buckle Insertion, covering contact transitions, constrained insertion, sustained surface interaction, and visually limited contact reasoning.
Figure 5: Real-world success rates. Per-task/stage success rate over 20 trials for each method; FR-VLA (ours) is shown in purple. Switch Cable Connection is evaluated in two stages, covering top Ethernet cable insertion and side power connector insertion. VLA-Adapter w/F denotes directly inputting time-aligned 6D force together with robot state.
Figure 6: Force admission trajectory in Switch Cable Connection. Keyframes, end-effector height, 6D wrench, and admission signal gtf over a rollout with two retractions before insertion is completed. Shaded regions denote inferred phases; admission stays positive throughout the inferred contact phase and becomes negative once the phase switches to free.
Task group
Task
Balanced acc.
Insertion
Gear Assembly
93.7%
Insertion
Switch Cable Connection
90.5%
Insertion
Seat Belt Buckle Insertion
88.5%
Sustained contact
Curved Surface Cleaning
80.6%
Sustained contact
Screw Tightening
86.2%
Table 1: Phase inference accuracy. Balanced accuracy after merging approach into free. Switch Cable Connection averages its two stages.
Variant
Gear
Switch-Top
Switch-Side
Cleaning
Screw
Buckle
Avg.
w/o force admission
60%
30%
15%
50%
25%
30%
35.0%
w/o force tokenization
70%
40%
20%
55%
20%
35%
40.0%
FR-VLA
90%
60%
35%
70%
40%
55%
58.3%
Table 2: Ablation results on force admission and force tokenization.
Method
Unseen gear
Target perturb.
VLA-Adapter w/F
60%
50%
FR-VLA
85%
75%
Table 3: Gear Assembly robustness over 20 trials per perturbation category.
Group 1
Group 2
Group 3
Group 4
4/10
6/10
5/10
5/10
Table 4: Switch Cable Connection under unseen initial cable positions, with 10 trials per group.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Robot platforms and camera setup. The two evaluation platforms, UR5e (left) and UR3e (right), each use the robot’s built-in 6D force/torque sensing. Both share the same camera configuration: a third-view Intel RealSense D455 camera and a wrist-mounted Intel RealSense D405 camera, highlighted on each platform.
Figure 8: Wrench magnitude gating on Gear Assembly. Six-dimensional wrench measurements and the admission signal of the magnitude gating baseline during a representative execution. The admission curve varies with the measured wrench amplitude.
Figure 9: Phase inference, force codes, and force admission across tasks. Frame borders and shaded regions denote predicted phases, and the code strip shows the selected VQ code kt . Additional trajectories are shown in Figure 10 .
Figure 10: Additional phase inference, force codes, and force admission trajectories. Continuation of Figure 9 ; frame borders, shaded regions, and the code strip use the same conventions.
Task
Manipulation segments
Training phases
Gear Assembly
1
Free, approach, contact
Switch Cable Connection
2
Free, approach, contact
Curved Surface Cleaning
1
Free, contact
Screw Tightening
1
Free, contact
Seat Belt Buckle Insertion
1
Free, contact
Appendix
Table 5: Phase supervision configuration. Contact detection uses shared thresholds; the training phase vocabulary differs by task.
Task
Free
Contact
Gear Assembly
64%
36%
Switch Cable Connection
53%
47%
Curved Surface Cleaning
44%
56%
Screw Tightening
56%
44%
Seat Belt Buckle Insertion
63%
37%
Appendix
Table 6: Distribution of offline motion phase labels. Relative prevalence of training labels after merging approach into free. Free includes approach; contact is unchanged. Switch Cable Connection is reported as one task rather than separating its two execution stages.
Figure 11: Association between force codes and interaction events. Each cell shows the lift P(event∣code)/P(event) on a logarithmic color scale; red indicates enrichment and blue indicates depletion.
Figure 12: Additional robustness test settings. (a) Target peg position changes. (b) Unseen gears. (c) Initial Ethernet cable position changes.