cs.CVNov 8, 2026

PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs

Authors: Ruimeng Liu, Denan Liang, Ruofei Bai, Fuming Zhang, Kuan Xu, Shenghai Yuan, Lihua Xie

Organizations: Nanyang Technological University, Singapore.

Abstract

Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.

Figures & tables

Explore similar work

CardsList
  1. SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

    Aug 1, 2026Yufei Zhang, Chenlu Zhan, Donghui Sun +2Vision-Language GroundingVLMs for Robotics

  2. Pointing-VLA: Typed Spatial Grounding Interfaces for Vision-Language-Action Manipulation

    Aug 24, 2026Xiwen Chen, Zelin Li, Zhiruo Zhou +3Spatial Grounding for VLAsVision-Language-Action Models