Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
Figures & tables
Figure 2: PhysMoE Architecture, Region-Based Supervision, and Continuous Coordinate Embedding . (a) Input RGB image and prompt. (b) Sequence tokenization. (c) Semi-MoE layers with shared self-attention and dynamically routed language/spatial experts ( FFNlang , FFNpix ). (d) Decoupled prediction heads for language tokens and 2D coordinates. (e) Region-based Mask-SDF supervision for continuous point prediction, penalizing exterior predictions while accommodating multi-peak affordance regions without mode averaging. (f) Continuous coordinate embedding via bilinear interpolation over visual positional embeddings Pvis and linear adapter projection for subsequent decoding steps.
Table 1: Comprehensive multi-task training mixture ( ∼ 12M samples) for PhysMoE . Datasets are structured across general multimodal cognition and embodied physical grounding. For composite suites ( e.g. , ShareRobot [ 24 ] ), we incorporate their respective domain subsets. Point and bounding-box annotations in affordance and contact tasks are converted into binary masks via SAM3 [ 26 , 41 ] for continuous Mask-SDF supervision.
Benchmark
Qwen3.5
Rynn1.0
RoboBrain
Ours
RoboBrain
Rynn1.1
Cosmos3
Embodied
HY
Ours
(4B)
(4B)
(4B)
(4B)
(8B)
(9B)
(8B)
(8B)
(30A3B)
(9B)
Part-Affordance
17.9
15.5
34.8
57.2
25.1
42.9
32.2
82.6
63.7
82.2
PIOBench
49.4
58.5
56.8
52.1
60.2
62.2
54.4
61.6
65.3
60.7
RefSpatial-Bench
39.7
56.8
53.3
53.8
54.9
62.0
44.4
52.4
53.4
55.2
RoboAfford
65.6
73.1
77.4
73.5
72.5
72.8
76.2
60.6
71.5
74.8
RoboRefit
81.0
61.0
83.5
82.0
82.4
81.9
55.4
77.2
88.2
82.0
Table 2: Comprehensive Evaluation on Coordinate-Based Spatial Grounding Tasks . Scores are on a 0 – 100 scale ( ↑ ). and denote the best and second-best results among parameter-comparable models, respectively. Model abbreviations: Rynn1.0 (RynnBrain1.0), Rynn1.1 (RynnBrain1.1), RoboBrain (RoboBrain2.5), Cosmos3 (Cosmos3-Nano), Embodied (Embodied-R1.5), HY (HY-Embodied-VLM-1.0).
Benchmark
Qwen3.5
Rynn1.0
RoboBrain
Ours
RoboBrain
Rynn1.1
Cosmos3
Embodied
HY
Ours
(4B)
(4B)
(4B)
(4B)
(8B)
(9B)
(8B)
(8B)
(30A3B)
(9B)
BLINK
78.8
82.2
78.0
75.5
80.9
85.0
82.4
77.8
87.3
80.6
CV-Bench
86.2
89.2
87.4
78.2
87.6
87.7
88.0
86.8
89.7
88.7
EmbSpatial
77.3
80.6
75.9
68.9
77.9
81.9
80.0
76.0
82.7
81.9
MindCube
35.0
55.5
34.0
46.2
29.3
81.1
32.8
27.9
70.0
79.0
SAT
69.3
65.3
64.0
71.3
66.0
76.7
54.0
60.7
78.0
78.0
Table 3: Evaluation on Text-Based Embodied Understanding & Reasoning Tasks . and denote the best and second-best results among parameter-comparable models, respectively.
Supervision Objective
Part-Afford.
RoboRefit
ShareRobot-Traj
RoboAfford
(a) Without Mask (Centroid ℓ1 )
55.5
79.2
85.5
72.3
(b) With Mask-SDF (Ours)
57.2
82.0
85.9
73.5
Table 4: Ablation Study on Supervision Objectives (Evaluated on 4B Model) . Comparison of single-point centroid regression versus our differentiable Mask-SDF supervision across affordance grounding, referring expressions, and trajectory prediction.
Model
Coord. Decoding
Throughput (samples/s) ↑
RynnBrain1.0-4B [ 30 ]
Autoregressive Tokens
5.28
RoboBrain2.5-4B [ 46 ]
Autoregressive Tokens
3.41
RynnBrain1.1-9B [ 30 ]
Autoregressive Tokens
2.61
EmbodiedR1.5-8B [ 61 ]
Autoregressive Tokens
2.81
RoboBrain2.5-8B [ 46 ]
Autoregressive Tokens
1.58
Ours-4B
Direct Regression
31.88 ( 6.04× )
Table 5: Pixel Inference Speed Comparison . Measured in samples/second ( samples/s↑ ). Speedups in parentheses are computed conservatively relative to the fastest competing baseline at each model scale.