Three-dimensional perception is critical for robotic manipulation, particularly for high-precision tasks, as recovering metric depth and precise 3D object positions from monocular RGB observations is inherently ill-posed. However, many Vision-Language-Action (VLA) models rely solely on RGB observations for perception. Leveraging recent advances in foundation models for stereo matching, we introduce ExStereo, a stereo module that augments pre-trained 2D VLAs with 3D perception. ExStereo reconstructs scene geometry from stereo image pairs and renders multi-view observations as an explicit stereo representation for stereo feature extraction. The action tokens from the action expert selectively attend to the resulting stereo tokens through our proposed action-stereo cross-attention mechanism, enabling the policy to generate robot actions conditioned on 3D scene information. To learn robust 3D representations, we introduce a mid-training stage before task-specific post-training, using a self-supervised learning objective on large-scale stereo data. We validate our approach by fine-tuning two publicly available VLAs, π0.5 and SmolVLA, and evaluate them in simulation and on a real-world bimanual PiPER platform. Across both settings, VLAs fine-tuned with ExStereo consistently outperform baselines, demonstrating the effectiveness of stereo perception for robotic manipulation. Our project website is at: https://exstereo-vla.github.io/ExStereo/.
Figures & tables
Fig. 1: ExStereo is a stereo module that enables 3D spatial perception in pre-trained, off-the-shelf Vision-Language-Action models. Given a stereo image pair, it constructs four orthogonal views of the scene with metric 3D coordinates. This explicit 3D representation allows the models to selectively retrieve visual features relevant to the action tokens through our proposed action-stereo cross-attention mechanism, improving robotic manipulation performance.
Fig. 2: Overview of ExStereo. ExStereo enhances pre-trained VLA models by leveraging a stereo camera and a pre-trained, frozen stereo foundation model [ 12 ] . Given a stereo image pair, the stereo model predicts a disparity map, which is converted into a metric depth map and 3D point cloud using the camera intrinsics and stereo baseline. We render the point cloud from four orthogonal viewpoints (Top, Front, Left, and Right) using z-buffering. Each view is rasterized at a resolution of 128×128 and contains depth, 3D coordinates, and a validity mask. The four views are horizontally tiled into a multi-view observation of size 128×512×5 , then processed by a Vision Transformer (ViT) [ 5 ] to extract stereo tokens z . We leverage z for action generation through our proposed action-stereo cross-attention, where action tokens a′ query z to selectively retrieve relevant 3D visual features while leaving the VLA’s prefix sequence and VLM weights unchanged. To learn robust 3D representations, we introduce mid-training with masked autoencoding (MAE), where the VLA reconstructs randomly masked multi-view patches from a large-scale stereo dataset while learning action-relevant features from incomplete observations. During VLA post-training, the ViT and action-stereo cross-attention block are frozen to prevent overfitting to task-specific demonstrations, while the MAE decoder is discarded.
Baselines
π0.5
SmolVLA
Task
ACT
DP3
StereoVLA
Base
ExStereo (w/o MT)
ExStereo
Base
ExStereo
Lift Pot
2.7
5.3
0.0
58.7
66.7
61.3
9.3
12.0
Adjust Bottle
10.7
2.7
53.3
76.0
89.3
96.0
58.7
96.0
Grab Roller
42.7
5.3
17.3
80.0
89.3
100.0
57.3
48.0
Open Laptop
0.0
5.3
2.7
61.3
72.0
92.0
2.7
44.0
Beat Block Hammer
0.0
4.0
0.0
25.3
45.3
70.7
1.3
24.0
TABLE I: Simulation Results. Per-task success rates (%) on ten bimanual RoboTwin tasks, averaged over three random seeds. We fine-tune two pre-trained VLAs, π0.5 and SmolVLA, with ExStereo (highlighted in green ) and compare them against their fine-tuned base policies. ACT (RGB), DP3 (point cloud), and StereoVLA serve as additional baselines. “w/o MT” denotes ExStereo without the mid-training stage. For each base VLA, the best result for each task is shown in bold . See Section V for more details.
Fig. 3: Simulation Tasks with Domain Randomization. The following tasks are used for policy evaluation. Top: Lift Pot , Adjust Bottle , Grab Roller , Open Laptop , and Beat Block Hammer ; Bottom: Click Alarmclock , Dump Bin Bigbin , Handover Mic , Press Stapler , and Stamp Seal . See the Appendix for details on domain randomization.
Method
Lift Pot
Open Laptop
VLM with Implicit Stereo Features
32.0
20.0
ASCA with Explicit Stereo Features (Ours)
66.7
72.0
Mid-training without MAE
57.3
90.7
Mid-training with MAE (Ours)
61.3
92.0
TABLE II: Ablation Studies of ExStereo. We compare the success rates (%) on two RoboTwin tasks between the VLM approach with implicit stereo features and our proposed action-stereo cross-attention (ASCA) approach with explicit stereo features (rows 1 and 2), as well as between mid-training without and with the masked autoencoding (MAE) objective (rows 3 and 4). All experiments use the pre-trained π0.5 as the base policy. Results are averaged over three training seeds.
Fig. 4: Real-World Rollouts with Multi-View Observations. Examples of successful π0.5 -ExStereo rollouts on two real-world tasks using the bimanual AgileX PiPER platform. Each column shows a trajectory over time, with each row containing the left stereo image and its corresponding multi-view observations (Top, Front, Left, Right) at a given timestep. The left column shows Close Fridge Door , in which the left arm closes the fridge door; the right column shows Lift Glass Bottles , in which both arms grasp and lift two glass bottles.
Method
Stack Two Bowls
Lift Glass Bottles
Close Fridge Door
π0.5 (Baseline)
2 / 20
5 / 20
9 / 20
StereoVLA
0 / 20
4 / 20
7 / 20
π0.5 -ExStereo w/o MT
9 / 20
11 / 20
16 / 20
π0.5 -ExStereo
13 / 20
18 / 20
19 / 20
TABLE III: Real-world Experiments. Results comparing ExStereo with baselines, with 20 trials per method and task combination. ‘w/o MT” denotes ExStereo without the mid-training stage.
Fig. 5: Point Cloud Reconstruction of Transparent Objects. The bottom-left image shows the point cloud captured by a RealSense RGB-D camera, which contains substantial noise and missing geometry due to the transparent objects. In contrast, the bottom-right image shows the point cloud reconstructed from a stereo image pair captured by a ZED Mini using Fast-FoundationStereo [ 12 ] , yielding cleaner and more geometrically consistent 3D points.
Hyper-parameter
Value
Point Cloud Reconstruction
Voxel downsampling
0.001 m
Radius outlier removal
30 points within 0.03 m
Table-plane RANSAC
threshold 0.01 m, n=3 , 1000 iterations
Canonical frame
table plane horizontal, surface at z=0
Orthographic Rendering
TABLE IV: ExStereo module, shared by π0.5 -ExStereo and SmolVLA-ExStereo.
Fig. 6: Real-World Setup. Our real-world platform with two AgileX PiPER arms. The pink dashed circle marks the ZED Mini 2 stereo camera used to capture the stereo pairs.
While Vision-Language-Action (VLA) models excel in generalist manipulation, they often lack fine-grained spatial awareness and show limited viewpoint robustness. This limitation largely stems from the reliance on pretrained RGB encoders, which lack explicit geometric cues and prioritize semantic alignment over geometric representation. We argue that effective visual representations for VLA models must jointly encode both semantic and geometric information. In this paper, we introduce StereoVLA, the first VLA model to incorporate rich geometric cues from large-scale synthetic stereo data. StereoVLA employs a Geometric-and-Semantic (GeoSem) vision encoder that extracts geometric cues from subtle stereo-view disparities for precise spatial perception, while simultaneously capturing semantic features from pixel observations to support language-conditioned manipulation. Additionally, we introduce two synergistic co-training objectives: Interaction-Region Depth Estimation for precise spatial reasoning, and Camera Parameter Estimation to implicitly align perception and action coordinate systems. Compared with baselines that employ various input modalities, StereoVLA achieves a 33.4% absolute gain in success rate in real-world experiments and demonstrates robustness to near-hemispheric camera perspectives. Project page: https://shengliangd.github.io/StereoVLA-Webpage.
Shengliang Deng, Mi Yan, Yixin Zheng +7
Galbot · The University of Hong Kong · CFCS, School of CS, Peking University +3
Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.
Evans Han, Yunfan Jiang, Yingke Wang +6
1Stanford University · 2Northwestern University · 3Lambda, Inc
We present GeomVLA, a Vision-Language-Action (VLA) model that unifies perception, latent scene motion prediction, and action generation within a shared robot-centric 3D coordinate frame. Our approach lifts pretrained VLM features into spatially grounded 3D scene tokens using depth and camera calibration, while retaining the semantic representations learned during VLM pretraining. We further introduce a 3D Scene Trajectory Denoiser, a task-conditioned module that learns a latent representation of how scene points are expected to move in 3D. Rather than executing the predicted trajectory as an open-loop plan, GeomVLA extracts intermediate motion tokens from the trajectory denoiser and uses them to condition a 3D flow-based action denoiser through geometry-aware attention. GeomVLA achieves state-of-the-art performance on CALVIN, competitive performance on LIBERO and RoboTwin2.0, and outperforms strong baselines in real-world manipulation settings without robot-action pretraining. Extensive ablations show that future-motion reasoning alone is insufficient: the primary gains are associated with maintaining geometric consistency among scene representation, motion prediction, and robot actions throughout the perception-to-action pipeline.