Vision-language-action (VLA) models enable robots to follow instructions in typed text, limiting broader deployment in natural speech. The traditional solution by prepending a speech recognition before VLA to convert speech into texts would incur additional latency and propagate errors. To address this, we present Speech-π0.5, which maps raw speech directly to motor commands without explicit textual transcriptions. Specifically, built on π0.5, we use a cross-modal projection to convert voice frames directly into audio tokens with significant response latency reduction. To further improve the accuracy, we adopt split LoRA adaptation with dedicated audio and task adapters, under a two-stage training. Speech-π0.5 achieves 3× lower response latency than ASR pipeline while maintaining competitive task success on LIBERO, and generalizes robustly to unseen accents where a single adapter degrades significantly.
Figures & tables
Figure 1. Speech- π0.5 : a frozen speech encoder’s output is projected into 32 VLM tokens. Split LoRA adapts backbone with separate audio and task adapters routed by token position. Architecture diagram showing audio processed by a frozen speech encoder, projected and pooled into the VLM prefix with split LoRA backbone adaptation.
Method
Spatial
Object
ASR pipeline + π0.5
95.6
95.0
Speech- π0.5 (train voices)
89.2
86.0
Speech- π0.5 (held-out accents)
85.4
84.8
Single LoRA (train voices)
69.0
69.0
Single LoRA (held-out accents)
68.4
38.0
Table 1. Task success (%) on LIBERO ( n=500 per condition). ASR pipeline: Whisper ASR + π0.5 text oracle. Single LoRA: standard single-adapter LoRA baseline.
Pipeline
First action (ms)
ASR pipeline: Whisper-v3 decode
216.7
+ π0.5 inference
107.7
= total
324.4
Speech- π0.5 (ours)
109.1
Response speedup
3.0 ×
Table 2. Response latency: the latency from speech input to robot action starting (H100, batch=1, 100 runs).