Organizations: The University of Tokyo, Japan · National Institute of Informatics, Japan · Institute of Science Tokyo, Japan · NII LLMC, Japan · The University of Osaka, Japan
Active robot vision requires controlling the camera to reveal task-relevant information that is hidden from the current viewpoint. For example, determining what is inside a box may require raising the camera and looking down into it. For viewpoint-dependent question answering, the challenge is to select camera motions that expose the visual evidence needed to answer the question. Although vision-language models (VLMs) can interpret observed images, selecting such motions requires anticipating the usefulness of unseen views. We quantify this usefulness as answerability, a VLM's estimate that a view suffices to answer the question, and present Rendering-Free Lookahead (RFL), a viewpoint-selection policy that ranks candidate camera motions by predicted future answerability. RFL transfers visual lookahead from deployment to offline training. At training, a privileged teacher renders candidate future views in 3D Gaussian Splatting (3DGS) scenes and uses a frozen VLM to compute one- and two-step answerability targets. Through two-stage distillation, a student learns to predict these action values from the question, recent visual observations, and a candidate camera motion. At deployment, RFL uses these predicted values to select camera motions without rendering future views. On 377 E3VS-Bench test episodes in unseen environments, RFL improves the mean judge score by 43% over a direct-action baseline using the same VLM. These results support learning camera-control policies from privileged visual lookahead for viewpoint-dependent question answering.
Figures & tables
Fig. 1: Rendering-free lookahead. RFL selects camera motions by predicted answerability, which rises as the agent closes in on a view that reveals the answer (right), with no rendering or generation at deployment.
Fig. 2: Overview of RFL. A privileged teacher renders each candidate action’s one- and two-step future views in 3DGS and scores them with the frozen VLM’s answerability probe, yielding the lookahead targets Q1 and Q2 ; the student learns to predict these values from the question, past views, and the action, requiring no rendering at deployment.
Fig. 3: Answerability tracks viewpoint search. Mean probed answerability vs. normalized progress on Gemini 3.0 Flash rollouts over the 231 validation episodes (bands: ±1 SE; dotted: mean at annotated goal views; Sec. V-C ).
Method
OS
OST
OA
CGS
SR
CNT
Avg. ↑
Coll. ↓
Proprietary
GPT-5.1
3.34
2.73
2.96
2.87
2.38
2.28
2.81
0.38
Gemini 3.0 Flash
3.14
3.09
2.67
2.47
2.25
1.96
2.62
0.43
Open-source
Cosmos3 (Planning)
2.83
2.18
2.38
2.33
2.25
1.92
2.36
0.50
Qwen3.6-27B
2.79
1.82
2.09
2.47
1.50
1.64
2.14
0.55
TABLE I: Main results on the E3VS-Bench test set. Green is best and light green is second best per column. OS = object search, OST = object state, OA = object attribute, CGS = context-guided search, SR = spatial reasoning, CNT = counting.
Fig. 4: Answerability along test rollouts for the agents of Table I (bands: ±1 SE; dashed: mean at goal views).
Figure 6
Fig. 8: Qualitative comparison on E3VS-Bench (RFL: green; Gemini 3.0 Flash: magenta). Full answers (abbreviated in the figure): middle, RFL: “The lid on the travel mug is beige.”; right, Gemini: “The text on the front of the plastic sack is not legible.”, RFL: “The text is not clear enough to be read.”
Fig. 9: Real-world experiments. Mean judge score per question for RFL, the direct base VLM (Base), and the GPT-5.1 agent. Each bar averages ten trials of one method on one question, i.e., 3 questions × 3 methods × 10 trials =90 trials.
Output
FT
Avg. ↑
Steps ↓
Rep. ↓
Actions
2.14
20.1
0.78
✓
2.47
24.9
0.87
Answerability
2.04
18.8
0.68
✓
2.71
17.2
0.62
TABLE II: Output and fine-tuning of the same VLM (Qwen3.6-27B) with single-view conditioning.
Real-world robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios. Existing view-robust Vision-Language-Action (VLA) policies tolerate such camera variations only when the camera extrinsics are explicitly provided, making them fragile and hard to use especially when view robustness is critical. We argue that the policy should not be told where the camera is, but rather figure it out by itself. To this end, we introduce Camera-Centric VLA (CamVLA), a new VLA model that decouples manipulation controls from camera geometry by predicting (i) a camera-centric end-effector action expressed in the local camera frame, and (ii) a 6-DoF hand-eye matrix relating cameras to the robot base. A deterministic geometric transformation composes the two predictions into a robot base-frame action. This disentangles how I should move in pose-independent camera-centric action generation from where I am looking from in camera-perspective geometric grounding. The resulting policy is calibration-free, depth-free, and single-view, requiring only a single monocular RGB image as the visual observation and task instruction at deployment. Evaluations in both simulation and real-world robot data show that CamVLA consistently improves success rates across diverse unseen viewpoints. Project page: https://alibaba-damo-academy.github.io/CamVLA/.
Wenhao Li, Xueying Jiang, Quanhao Qian +4
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We call this capability view planning, requiring (1)understanding how a single action transforms the view, and (2)composing many such transformations across multi-turn plans to identify a target view. We probe both abilities in our proposed ViewSuite, a 3D point-cloud environment on real ScanNet scenes. Across 13 frontier VLMs, a critical planning gap emerges: they possess basic view-action knowledge but fail to compose it across multi-turn plans, with the gap widening as viewpoint distance grows. To close this gap, we propose an iterative framework that alternates self-exploration with view graph distillation. The key insight is that all exploration trajectories, regardless of their outcome, collectively form a view graph that compactly captures how viewpoints connect across a scene. Distilling this graph into diverse supervised tasks reshapes the policy distribution and overcomes the sparse rewards that stall pure RL. This improves Qwen2.5-VL-7B from 2.5% to 47.8% on interactive view planning, surpassing GPT-5.4 Pro (18.5%) and Gemini 3.1 Pro (21.4%). Self-exploration emerges as a promising path toward VLMs that can actively reason and plan in 3D space. Code and Data are at https://viewsuite.github.io.
Kangrui Wang, Linjie Li, Zhengyuan Yang +7
Northwestern University · University of Washington · Microsoft +2
Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, language-conditioned camera motion remains comparatively underexplored as a first-class action. We formulate language-conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. This task is inherently non-trivial: viewpoint changes are driven by latent perceptual intentions, and a valid motion may operate at different semantic granularity, from entering a room to looking around a corner, inspecting a visible object, or revealing an occluded detail. To model this structure, we mine multi-intention camera-motion supervision from egocentric video, pairing plausible intents and observation-gain descriptions with relative SE(3) target poses. We propose LIME, a vision-language camera-motion generator that combines an auto-regressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across experiments and downstream robotic tasks, we show that LIME can learn to actively choose camera poses from passive human video, turning ordinary egocentric recordings into supervision for intent-aware active perception.