Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
Figures & tables
Figure 2: PhysMoE Architecture, Region-Based Supervision, and Continuous Coordinate Embedding . (a) Input RGB image and prompt. (b) Sequence tokenization. (c) Semi-MoE layers with shared self-attention and dynamically routed language/spatial experts ( FFNlang , FFNpix ). (d) Decoupled prediction heads for language tokens and 2D coordinates. (e) Region-based Mask-SDF supervision for continuous point prediction, penalizing exterior predictions while accommodating multi-peak affordance regions without mode averaging. (f) Continuous coordinate embedding via bilinear interpolation over visual positional embeddings Pvis and linear adapter projection for subsequent decoding steps.
Table 1: Comprehensive multi-task training mixture ( ∼ 12M samples) for PhysMoE . Datasets are structured across general multimodal cognition and embodied physical grounding. For composite suites ( e.g. , ShareRobot [ 24 ] ), we incorporate their respective domain subsets. Point and bounding-box annotations in affordance and contact tasks are converted into binary masks via SAM3 [ 26 , 41 ] for continuous Mask-SDF supervision.
Benchmark
Qwen3.5
Rynn1.0
RoboBrain
Ours
RoboBrain
Rynn1.1
Cosmos3
Embodied
HY
Ours
(4B)
(4B)
(4B)
(4B)
(8B)
(9B)
(8B)
(8B)
(30A3B)
(9B)
Part-Affordance
17.9
15.5
34.8
57.2
25.1
42.9
32.2
82.6
63.7
82.2
PIOBench
49.4
58.5
56.8
52.1
60.2
62.2
54.4
61.6
65.3
60.7
RefSpatial-Bench
39.7
56.8
53.3
53.8
54.9
62.0
44.4
52.4
53.4
55.2
RoboAfford
65.6
73.1
77.4
73.5
72.5
72.8
76.2
60.6
71.5
74.8
RoboRefit
81.0
61.0
83.5
82.0
82.4
81.9
55.4
77.2
88.2
82.0
Table 2: Comprehensive Evaluation on Coordinate-Based Spatial Grounding Tasks . Scores are on a 0 – 100 scale ( ↑ ). and denote the best and second-best results among parameter-comparable models, respectively. Model abbreviations: Rynn1.0 (RynnBrain1.0), Rynn1.1 (RynnBrain1.1), RoboBrain (RoboBrain2.5), Cosmos3 (Cosmos3-Nano), Embodied (Embodied-R1.5), HY (HY-Embodied-VLM-1.0).
Benchmark
Qwen3.5
Rynn1.0
RoboBrain
Ours
RoboBrain
Rynn1.1
Cosmos3
Embodied
HY
Ours
(4B)
(4B)
(4B)
(4B)
(8B)
(9B)
(8B)
(8B)
(30A3B)
(9B)
BLINK
78.8
82.2
78.0
75.5
80.9
85.0
82.4
77.8
87.3
80.6
CV-Bench
86.2
89.2
87.4
78.2
87.6
87.7
88.0
86.8
89.7
88.7
EmbSpatial
77.3
80.6
75.9
68.9
77.9
81.9
80.0
76.0
82.7
81.9
MindCube
35.0
55.5
34.0
46.2
29.3
81.1
32.8
27.9
70.0
79.0
SAT
69.3
65.3
64.0
71.3
66.0
76.7
54.0
60.7
78.0
78.0
Table 3: Evaluation on Text-Based Embodied Understanding & Reasoning Tasks . and denote the best and second-best results among parameter-comparable models, respectively.
Supervision Objective
Part-Afford.
RoboRefit
ShareRobot-Traj
RoboAfford
(a) Without Mask (Centroid ℓ1 )
55.5
79.2
85.5
72.3
(b) With Mask-SDF (Ours)
57.2
82.0
85.9
73.5
Table 4: Ablation Study on Supervision Objectives (Evaluated on 4B Model) . Comparison of single-point centroid regression versus our differentiable Mask-SDF supervision across affordance grounding, referring expressions, and trajectory prediction.
Model
Coord. Decoding
Throughput (samples/s) ↑
RynnBrain1.0-4B [ 30 ]
Autoregressive Tokens
5.28
RoboBrain2.5-4B [ 46 ]
Autoregressive Tokens
3.41
RynnBrain1.1-9B [ 30 ]
Autoregressive Tokens
2.61
EmbodiedR1.5-8B [ 61 ]
Autoregressive Tokens
2.81
RoboBrain2.5-8B [ 46 ]
Autoregressive Tokens
1.58
Ours-4B
Direct Regression
31.88 ( 6.04× )
Table 5: Pixel Inference Speed Comparison . Measured in samples/second ( samples/s↑ ). Speedups in parentheses are computed conservatively relative to the fastest competing baseline at each model scale.
Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified vision-language model with two variants, 4B and 35B-A3B, that brings together embodied cognition and multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning embodied and multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises embodied capability injection, separate reinforcement learning of embodied and multimodal-agent experts, and multi-teacher on-policy distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both embodied and agent benchmarks, as well as on autonomous-driving and embodied-navigation tasks. For edge deployment, visual token compression, W4A8 quantization, and hardware-software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: https://machembodied.com/ME-Brain/ME-VLM.html Code Repository: https://github.com/MachEmbodied/ME-VLM
Affordance grounding aims to localize the functional region for interaction, such as the handle to grasp or the button to press, rather than the whole object. This makes it more challenging than generic visual grounding because the target region is smaller, more ambiguous, and more dependent on task context, especially for compact vision-language models (VLMs) used in embodied settings. Recent sequence-level supervision and reinforcement learning improve coordinate prediction quality, yet compact autoregressive VLMs still lack reliable affordance-aware visual focus before coordinate generation: the model can produce better coordinate tokens while its cross-modal attention remains diffuse and weakly anchored to the true affordance evidence. To address it, we propose SpatialAfford, a two-stage framework that first aligns attention to the ground-truth affordance region through Spatial Attention Alignment (SAA), then refines coordinate prediction with Spatial-Aware GRPO. By explicitly teaching the model where to look before optimizing where to ground, SpatialAfford turns affordance grounding from a purely output-constrained objective into attention-grounded spatial reasoning. Across ShareRobot-Bench, ReasonAff, and PartAfford, SpatialAfford consistently improves affordance grounding, with a compact 4B model outperforming stronger 7B+ baselines.
Yufei Zhang, Chenlu Zhan, Donghui Sun +2
Zhejiang University · Vivo Mobile Communication Co., Ltd
Vision-language-action (VLA) models often expose spatial grounding through autoregressive text coordinates or opaque action tokens, creating brittle interfaces between multimodal reasoning and robot execution. We present Pointing-VLA, a typed hidden-state spatial readout built on Embodied-R1. Geometry-specific heads predict normalized points, object-functional grounding (OFG) heatmaps, and visual trajectories without serializing geometry as text. For the evaluated Bridge/WidowX and physical pick-place deployments, an explicit execution contract assigns PICK to source-conditioned OFG and PLACE to Pointing, providing direct stage-aligned spatial targets. Pointing-VLA achieves SOTA performance on Bridge/WidowX, averaging 72.9% across the evaluated four-task set without Bridge-specific finetuning under collision-enabled CuRobo execution. Pointing and OFG show complementary strengths across native and cross-dataset evaluations. The OFG/contact readout transfers to NORA-1.5, preserving or improving success while reducing recorded controller time by more than 20×; typed heads are also 6.68--6.90× faster than Embodied-R1 text decoding on a shared external suite. When integrated as spatial guidance for a π0.5 action policy, Pointing-VLA raises autonomous real-robot success from 52.7% to 80.7% across three visual contexts. These results establish typed spatial readouts as an efficient, inspectable interface between embodied reasoning and robot execution.
Xiwen Chen, Zelin Li, Zhiruo Zhou +3
SIGS, Tsinghua University, China · Shanghai Jiao Tong University, China · Wuhan University of Technology, China +2