Three-dimensional perception is critical for robotic manipulation, particularly for high-precision tasks, as recovering metric depth and precise 3D object positions from monocular RGB observations is inherently ill-posed. However, many Vision-Language-Action (VLA) models rely solely on RGB observations for perception. Leveraging recent advances in foundation models for stereo matching, we introduce ExStereo, a stereo module that augments pre-trained 2D VLAs with 3D perception. ExStereo reconstructs scene geometry from stereo image pairs and renders multi-view observations as an explicit stereo representation for stereo feature extraction. The action tokens from the action expert selectively attend to the resulting stereo tokens through our proposed action-stereo cross-attention mechanism, enabling the policy to generate robot actions conditioned on 3D scene information. To learn robust 3D representations, we introduce a mid-training stage before task-specific post-training, using a self-supervised learning objective on large-scale stereo data. We validate our approach by fine-tuning two publicly available VLAs, π0.5 and SmolVLA, and evaluate them in simulation and on a real-world bimanual PiPER platform. Across both settings, VLAs fine-tuned with ExStereo consistently outperform baselines, demonstrating the effectiveness of stereo perception for robotic manipulation. Our project website is at: https://exstereo-vla.github.io/ExStereo/.
Figures & tables
Fig. 1: ExStereo is a stereo module that enables 3D spatial perception in pre-trained, off-the-shelf Vision-Language-Action models. Given a stereo image pair, it constructs four orthogonal views of the scene with metric 3D coordinates. This explicit 3D representation allows the models to selectively retrieve visual features relevant to the action tokens through our proposed action-stereo cross-attention mechanism, improving robotic manipulation performance.
Fig. 2: Overview of ExStereo. ExStereo enhances pre-trained VLA models by leveraging a stereo camera and a pre-trained, frozen stereo foundation model [ 12 ] . Given a stereo image pair, the stereo model predicts a disparity map, which is converted into a metric depth map and 3D point cloud using the camera intrinsics and stereo baseline. We render the point cloud from four orthogonal viewpoints (Top, Front, Left, and Right) using z-buffering. Each view is rasterized at a resolution of 128×128 and contains depth, 3D coordinates, and a validity mask. The four views are horizontally tiled into a multi-view observation of size 128×512×5 , then processed by a Vision Transformer (ViT) [ 5 ] to extract stereo tokens z . We leverage z for action generation through our proposed action-stereo cross-attention, where action tokens a′ query z to selectively retrieve relevant 3D visual features while leaving the VLA’s prefix sequence and VLM weights unchanged. To learn robust 3D representations, we introduce mid-training with masked autoencoding (MAE), where the VLA reconstructs randomly masked multi-view patches from a large-scale stereo dataset while learning action-relevant features from incomplete observations. During VLA post-training, the ViT and action-stereo cross-attention block are frozen to prevent overfitting to task-specific demonstrations, while the MAE decoder is discarded.
Baselines
π0.5
SmolVLA
Task
ACT
DP3
StereoVLA
Base
ExStereo (w/o MT)
ExStereo
Base
ExStereo
Lift Pot
2.7
5.3
0.0
58.7
66.7
61.3
9.3
12.0
Adjust Bottle
10.7
2.7
53.3
76.0
89.3
96.0
58.7
96.0
Grab Roller
42.7
5.3
17.3
80.0
89.3
100.0
57.3
48.0
Open Laptop
0.0
5.3
2.7
61.3
72.0
92.0
2.7
44.0
Beat Block Hammer
0.0
4.0
0.0
25.3
45.3
70.7
1.3
24.0
TABLE I: Simulation Results. Per-task success rates (%) on ten bimanual RoboTwin tasks, averaged over three random seeds. We fine-tune two pre-trained VLAs, π0.5 and SmolVLA, with ExStereo (highlighted in green ) and compare them against their fine-tuned base policies. ACT (RGB), DP3 (point cloud), and StereoVLA serve as additional baselines. “w/o MT” denotes ExStereo without the mid-training stage. For each base VLA, the best result for each task is shown in bold . See Section V for more details.
Fig. 3: Simulation Tasks with Domain Randomization. The following tasks are used for policy evaluation. Top: Lift Pot , Adjust Bottle , Grab Roller , Open Laptop , and Beat Block Hammer ; Bottom: Click Alarmclock , Dump Bin Bigbin , Handover Mic , Press Stapler , and Stamp Seal . See the Appendix for details on domain randomization.
Method
Lift Pot
Open Laptop
VLM with Implicit Stereo Features
32.0
20.0
ASCA with Explicit Stereo Features (Ours)
66.7
72.0
Mid-training without MAE
57.3
90.7
Mid-training with MAE (Ours)
61.3
92.0
TABLE II: Ablation Studies of ExStereo. We compare the success rates (%) on two RoboTwin tasks between the VLM approach with implicit stereo features and our proposed action-stereo cross-attention (ASCA) approach with explicit stereo features (rows 1 and 2), as well as between mid-training without and with the masked autoencoding (MAE) objective (rows 3 and 4). All experiments use the pre-trained π0.5 as the base policy. Results are averaged over three training seeds.
Fig. 4: Real-World Rollouts with Multi-View Observations. Examples of successful π0.5 -ExStereo rollouts on two real-world tasks using the bimanual AgileX PiPER platform. Each column shows a trajectory over time, with each row containing the left stereo image and its corresponding multi-view observations (Top, Front, Left, Right) at a given timestep. The left column shows Close Fridge Door , in which the left arm closes the fridge door; the right column shows Lift Glass Bottles , in which both arms grasp and lift two glass bottles.
Method
Stack Two Bowls
Lift Glass Bottles
Close Fridge Door
π0.5 (Baseline)
2 / 20
5 / 20
9 / 20
StereoVLA
0 / 20
4 / 20
7 / 20
π0.5 -ExStereo w/o MT
9 / 20
11 / 20
16 / 20
π0.5 -ExStereo
13 / 20
18 / 20
19 / 20
TABLE III: Real-world Experiments. Results comparing ExStereo with baselines, with 20 trials per method and task combination. ‘w/o MT” denotes ExStereo without the mid-training stage.
Fig. 5: Point Cloud Reconstruction of Transparent Objects. The bottom-left image shows the point cloud captured by a RealSense RGB-D camera, which contains substantial noise and missing geometry due to the transparent objects. In contrast, the bottom-right image shows the point cloud reconstructed from a stereo image pair captured by a ZED Mini using Fast-FoundationStereo [ 12 ] , yielding cleaner and more geometrically consistent 3D points.
Hyper-parameter
Value
Point Cloud Reconstruction
Voxel downsampling
0.001 m
Radius outlier removal
30 points within 0.03 m
Table-plane RANSAC
threshold 0.01 m, n=3 , 1000 iterations
Canonical frame
table plane horizontal, surface at z=0
Orthographic Rendering
TABLE IV: ExStereo module, shared by π0.5 -ExStereo and SmolVLA-ExStereo.
Fig. 6: Real-World Setup. Our real-world platform with two AgileX PiPER arms. The pink dashed circle marks the ZED Mini 2 stereo camera used to capture the stereo pairs.