FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs
Organizations: Stanford University · Princeton University
Abstract
We present FoveaVLA, a Vision-Language-Action (VLA) model that replaces the wrist camera with learned foveation actions. Wrist cameras provide close-up detail but restrict the view to the end-effector's surroundings. External cameras see the whole scene, but the policy receives them at low resolution, so the fine detail needed for manipulation is lost. FoveaVLA recovers this detail by predicting, alongside each action chunk, the interaction region to observe at the next policy query. The policy then receives crops of these regions, taken from the original-resolution images, together with the full views. We build FoveaVLA on in two configurations, Single and Dual, using one or two external stereo cameras with full and foveal RGB and depth inputs, and post-train both on DROID with RoboInter interaction annotations. Across four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, and FoveaVLA-Dual achieves 77.1%, versus 52.1% for the wrist-equipped . Both match the baseline on plain pick-and-place and improve success with distractors, cloth manipulation, and far-away objects.
Figures & tables
| Method | Pick-and-Place (plain) | Pick-and-Place (w/ Distractors) | Fold/Unfold | Move (far objects) | Overall |
|---|---|---|---|---|---|
| [ 23 ] | 100.0 0.0 | 25.0 13.1 | 41.7 14.9 | 41.7 14.9 | 52.1 7.3 |
| -Blackout | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
| -Single-External | 16.7 11.2 | 0.0 0.0 | 0.0 0.0 | 8.3 8.3 | 6.3 3.5 |
| -Dual-External | 50.0 15.1 | 16.7 11.2 | 16.7 11.2 | 16.7 11.2 | 25.0 6.3 |
| ECoT-inspired [ 49 ] | 66.7 14.2 | 16.7 11.2 | 16.7 11.2 | 25.0 13.1 | 31.3 6.8 |
| -Single-External-RGB-D | 8.3 8.3 | 8.3 8.3 | 0.0 0.0 | 8.3 8.3 | 6.3 3.5 |
| Task | -DROID | FoveaVLA-Dual (ours) |
|---|---|---|
| Pick-v1.5 | 18.1 | 33.7 |
| Pick-v2-Classic | 6.4 | 9.7 |
| Pick-v2-Filament | 7.0 | 10.6 |
| Pick-v2-RandCam | 8.0 | 5.5 |
| Pick&Place-v2 | 12.0 | 17.0 |
| Pick&Place-NextTo-v2 | 10.3 | 14.9 |
| Method | FT | Frame | [email protected] | mIoU | Wait (ms) |
|---|---|---|---|---|---|
| -DROID | No | 0.002 | 0.009 | 0 | |
| PaliGemma + PG | No ∗ | 0.132 | 0.207 | 427 | |
| SAM3 + PG | No ∗ | 0.305 | 0.347 | 690 | |
| YOLO11s | Yes | 0.488 | 0.492 | 29 | |
| FoveaVLA (ours) | Yes | 0.483 | 0.465 | 0 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Primitive skills | Required components |
|---|---|
| Pick, push, pull, press, twist | Gripper and object/tool |
| Transfer, place, slide, fold, pour, insert, throw, manipulate, shake, strike | Gripper, object/tool, and target region |
| Setting | Value |
|---|---|
| Initialization | base checkpoint |
| Training updates / global batch | / |
| Training hardware | NVIDIA H100 GPUs |
| Optimizer | AdamW |
| Adam / | / |
| Weight decay / gradient norm clip | / |
| Tasks | Renderer | External-camera layout |
|---|---|---|
| Pick-v1.5; Open-v1; Close-v1 | Classic | Original external view plus fixed right-shoulder view |
| Pick-v2-Classic | Classic | Left/right shoulder views |
| Pick-v2-Filament; all three Pick&Place-v2 tasks | Filament | Left/right shoulder views |
| Pick-v2-RandCam | Filament | Randomized views on opposite sides |
| Trial | Objects and initial positions | Language instruction |
|---|---|---|
| 0 | S: red marker (BR); T: blue bowl (FR) | Place the marker in the blue bowl |
| 1 | S: black marker (FL); T: blue bowl (BL) | Put the pen in the bowl |
| 2 | S: red marker (BL); T: orange bowl (FR) | Place the red marker in the orange bowl |
| 3 | S: small cube (FL); T: orange bowl (FR) | Place the small block in the bowl |
| 4 | S: blue cube (FR); T: blue bowl (BL) | Place the block in the bowl |
| 5 | S: small cube (BR); T: blue bowl (FR) | Place the small block in the blue bowl |
| Trial | Objects and initial positions | Language instruction |
|---|---|---|
| 12 | S: blue cube (BL); R: green cup (BR) | Move the cube next to the cup |
| 13 | S: small cube (FR); R: white cup (FL) | Move the cube next to the white mug |
| 14 | S: blue cube (BR); R: silver cup (BL) | Move the block next to the cup |
| 15 | S: blue cube (FL); R: white cup (BR) | Move the blue block near the white mug |
| 16 | S: silver cup (BL); R: yellow bowl (BR) | Move the cup near the yellow bowl |
| 17 | S: silver cup (FL); R: orange bowl (BL) | Move the silver mug near the bowl |
| Trial | Objects and initial positions | Language instruction |
|---|---|---|
| 24 | T: blue towel (FL) | Fold the blue towel |
| 25 | T: grey towel (FR) | Fold the grey towel |
| 26 | T: pink towel (BL) | Fold the pink towel |
| 27 | T: blue towel (FL); D: grey towel (BL) | Fold the blue towel |
| 28 | T: grey towel (FR); D: pink towel (BR) | Fold the grey towel |
| 29 | T: pink towel (BL); D: blue towel (FL) | Fold the pink towel |
| Trial | Objects and initial positions | Language instruction |
|---|---|---|
| 36 | S: red marker (BL); T: blue bowl (FM); D: blue marker (BR) | Place the red marker in the blue bowl |
| 37 | S: blue marker (BR); T: orange bowl (FM); D: black marker (BL) | Place the blue marker in the orange bowl |
| 38 | S: black marker (BL); T: yellow bowl (FM); D: red marker (BR) | Place the black marker in the yellow bowl |
| 39 | S: red marker (BR); T: blue bowl (FL); D: blue marker (BL); orange bowl (FR) | Place the red marker in the blue bowl |
| 40 | S: blue marker (BL); T: orange bowl (FR); D: black marker (BR); yellow bowl (FL) | Place the blue marker in the orange bowl |
| 41 | S: black marker (BR); T: yellow bowl (FL); D: red marker (BL); blue bowl (FR) | Place the black marker in the yellow bowl |