A Phenomenology of Neural Geometry in Vision-Language-Action Models
Authors: Mozes Jacobs, Aiden Swann, Mathilde Papillon, Siddharth Boppana, Fenil R. Doshi, Usha Bhalla, Thomas Icard, Matt Feiszli, Thomas McGrath, Vasudev Shyam, Demba Ba, Leon Bergen, Nina Miolane, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Matthew Kowal, Thomas Fel
Organizations: Goodfire · Harvard University · Stanford University · University of California, Santa Barbara
Robotic manipulation is inherently geometric, but what shape does the world take inside the policy that performs it? Prior work on vision-language-action (VLA) policies establishes that task-relevant variables are linearly readable from activations, yet readability certifies only that a variable is present and says nothing about the shape its values trace, which may be curved and multi-dimensional. Using controlled sweeps, natural rollouts, linear probes, and activation interventions, we build a taxonomy of seven task-relevant concepts in VLA models. We find that realized geometry is predominantly multi-dimensional and is shaped by the constraints under which the policy acts. Physical joint limits, for example, leave wrist orientation an open arc rather than a circle, and distance to the target, a single scalar, is realized in two orthogonal subspaces, a coarse ruler for the approach and a fine ruler for the grasp. The same geometries reappear in two further VLA models on real-world robot data, indicating that they are neither model-specific nor artifacts of simulation. We demonstrate how the findings are actionable by steering along the geometry a concept realizes. Overall, this work lays out an initial phenomenology of realized geometry in VLA policies and demonstrates that a concept's shape carries as much importance as its readability.
Figures & tables
Figure 1: Control variables realize distinct geometries in a VLA policy. MolmoBot ( Deshpande et al., 2026 ) activations over many states (clouds) and controlled sweeps varying one physical quantity (traces). Clouds and traces are activations projected onto a low-dimensional basis (principal components for the gripper and wrist; the span of a linear probe’s weights for target position); for distance, the activations along the two ruler directions are plotted against the true distance (z axis). ( Gripper ) Open and closed states from natural rollouts separate along one axis; arrows show interventions reversing the command. ( Wrist ) Rotation sweeps pooled over 64 houses trace an open arc, not the circle unconstrained rotation would give. ( Target position ) A structured trajectory within the cloud of teleported target placements. ( Distance ) Two orthogonal rulers, coarse for approach and fine for grasp.
Figure 2: MolmoBot’s VLM encodes its end-effector (layer 20) and the target object (layer 21) in separate, approximately Euclidean 3-D subspaces. Top: (a) held-out per-axis error relative to gripper width; (b) held-out R2 vs. readout rank; (c) singular values of the rank-3 readout; (d) residual of a linear map from world to activation displacements across layers (solid: general linear; dashed: orthogonal; reference line: no linear relation); (e) curvature of straight world paths inside each subspace vs. distance to the gripper, compared with a random 3-D projection. Middle, bottom: five gripper paths through the workspace (middle) and five target paths across its support surface (bottom), shown in world coordinates (upper) and projected into the corresponding subspace (lower). Subspaces are fit with the tested house held out.
Figure 3: MolmoBot’s VLM encodes distance to the target in nearly orthogonal near- and far-range codes (layer 21). (a) Near ( <15 cm) and far (15–70 cm) probe R2 after erasing the top- k directions of the other probe, computed against each range’s own variance. Erasing far directions leaves near decoding intact while far decoding collapses, and vice versa. The intact ( k=0 ) probes have errors of ∼1.3 cm (near) and ∼3.6 cm (far). (b) Near R2 ( <c ) vs. far R2 ( ≥c ) as the cutoff c is swept. Clipped (‘round to c’) probes match probes fit separately on each range and outperform a single full-range probe. (c) Cosine between the near- and far-probe weight vectors as the cutoff c is swept, for range-split and clipped (‘round to c’) pairs (fold mean ± s.d.). For c≲30 cm, both cosines stay below the cosine between two copies of the same probe fit on different halves of the data (dashed). On synthetic activations that encode distance along a single direction, the same probes are nearly parallel for c≳15 cm (dotted). (d) Activations plotted by their meter- and centimeter-probe readouts, colored by true distance in meters (left) and centimeters (right). Far samples vary along the meter axis while the centimeter readout saturates, and near samples do the opposite, tracing an L.
Figure 4: MolmoBot’s VLM encodes contact with the target along a direction that survives off-schedule contact (layer 24). (a) Natural rollouts projected onto their first three principal components, colored by contact; the two states separate cleanly. (b) An LDA probe trained on natural rollouts decodes contact with 91.3% accuracy. When we force an early touch or an induced slip by teleporting the object, the same probe still decodes contact well above chance.
Figure 5: MolmoBot’s action expert encodes task phase, but infers it from motion, so pauses are read as phase transitions (layer 24). Top: an example pick-and-place trajectory and its path through the phase subspace. Bottom: (a) LDA projection of natural rollouts, colored by phase. (b) First three principal components of natural activations, colored by true phase; the example trajectory is colored by the LDA probe readouts. (c) Freezing the arm shifts the decoded phase toward grasp during reach; frozen place landmarks are most often still decoded as place. Right: most frequent decoded phases when frozen (top two for reach and place). (d) Change in the fraction of executed-chunk steps commanding a closed gripper (frozen − natural; mean ± 95% CI across landmarks). Frozen reach and grasp moments close the gripper, while frozen place moments open it (Fig. 19 ).
Figure 6
Figure 6: The action expert encodes wrist orientation as an open arc, and edits to it act as beliefs (layer 9). We sweep the wrist’s roll joint through its full range ( −172.8∘ to +172.8∘ ), holding other joints fixed, and record activations at each angle (camera views in Fig. 20 ). (a) The sweep traces an open curve: its endpoints, nearly the same physical pose, lie far apart. (b) An autoencoder maps activations onto an exact circle. (c) A single ±20∘ or ±40∘ edit produces a decoded rotation opposing the written angle (dots: seven held-out scenes; ticks: means; dashed diagonal: the response a command would produce). (d) Holding the edit at every step compounds the corrections, rotating the wrist opposite the written angle by about its magnitude (vertical lines mark action chunks).
Figure 7: Gripper control is one-dimensional (AE, layer 32). (a) The first principal component of the pullback steering vectors captures ∼70% of their variance. (b) One principal component of the pullback steering vectors decodes the commanded gripper aperture nearly as well as the full activation. (c) Natural-rollout activations colored by gripper state; the steering vectors point from each cluster toward the other. (d) One component flips the gripper but leaks into the arm joints relative to using the full 768-dimensional steering vector; more components remove this excess leakage.
Figure 8: The geometry recovered in MolmoBot largely reappears in π0.5 and MolmoAct2 on real-world DROID data. Procrustes similarity between each pair of models is highest for the gripper and lowest for wrist orientation, and all values exceed a shuffled null ( 0.00 – 0.02 for gripper and distance; 0.11 – 0.18 for wrist and EE position). ( Gripper ) Pullback steering vectors (379 per model) on their top three principal components, colored by whether they close (blue) or open (red) the gripper. ( Wrist orientation ) Activations on a (cosθ,sinθ) probe, colored by angle. ( EE position ) Activations on the (x,y,z) probe. ( Distance to grasp ) The real-world proxy for distance to target: activations on the meter and centimeter rulers of Fig. 3 , with far (cool) and near (warm) points spreading along different rulers. Episode trajectories are overlaid in the last three columns.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The action expert (layer 12) also maintains a rank-3 EE position subspace. Panels as in the top and middle rows of Figure 2 , for EE position read from the action expert, which receives proprioception; the paths are from a single scene excluded when fitting the subspace. The readout is near-exact and the paths are recovered almost without distortion. The gap between the general-linear and orthogonal residuals shows the subspace is a scaled, not isometric, copy of world coordinates.
Figure 10: The action expert (layer 22) also maintains a rank-3 target object position subspace. Panels as in Figure 9 , for the target object’s position; the paths are from a single scene excluded when fitting the subspace. The action expert receives the object only through cross-attention to the VLM, and it carries target position with larger error than the VLM (Figure 2 ), recovering the paths with more distortion.
Figure 11: Within a scene, target object position is computed on the VLM’s instruction and final tokens between layers 12 and 21 and appears in the action expert from layer 18; EE position is present from the first layer in both. Held-out R2 of the rank-3 ridge readout of end-effector position (left) and target object position (right) at every layer, scored after removing each held-out house’s mean from the true and predicted positions, so only position changes within a scene count; one line per VLM token pool, the action expert dashed, rings on the readouts of Figure 2 , values below the axis floor clipped. The target’s height is constant within a scene, so the target panel’s variance is horizontal and any within-scene variation in the predicted height counts as error; scores fall below zero where a readout’s predictions vary within a scene without tracking the object. On the instruction and final tokens the target read stays at or below zero until layer 12 and peaks at layers 18–21, while the image patches carry little of it at any layer. The action expert tracks EE position near-perfectly at every layer and the target only from layer 18 on.
Figure 12: Distance is compressive within the near range and closest to linear beyond 30 cm, and each ruler reads only its own range (VLM, layer 21). (Left) Held-out mean absolute error per bin of true distance for a single full-range ridge probe fit to d , logd , or 1/d and mapped back to metres; the compressive links have the lower error below 15 cm and the linear link in the farthest bins. (Right) Held-out R2 against true distance for the centimeter ruler and the meter ruler of Figure 3 , scored separately on states below and above that cutoff; negative (saturated) values are drawn at zero and printed. Each ruler saturates outside its own range, as does a single unclipped metre-scale probe (not shown) on the near-range states.
Figure 13: The action expert (layer 35) reproduces the two-subspace distance code. Same store, folds, and constructions as Figure 3 , read from the action expert instead of the VLM. (Top left) Held-out R2 of the near and far decodes as far-axis (solid) or near-axis (dashed) directions are erased; each decode drops only when its own axis is removed. (Top right) Held-out R2 of the centimeter and meter rulers on each side of the 15 cm cutoff; each ruler scores only its own range, and bars below zero are drawn at zero with the value printed. (Bottom) The two ruler readouts plotted against each other, colored by true distance on the meter (left) and centimeter (right) scales; the meter ruler resolves the far arm and the centimeter ruler the near arm, as in the VLM cell.
Figure 14: Direction to target object is observable with one probe, in contrast to the two distance rulers. Top: held-out decoded directions (VLM layer 21), averaged in 6 cm cells in a target-centered frame; arrow length is cell coherence. Decoded directions point toward the target throughout the workspace (median angular error 11.7∘ vs. a 33.1∘ floor); within 15 cm, per-read error increases but cell means still point toward the target (coherence 0.98 beyond 40 cm vs. 0.77 inside 7.5 cm). Bottom left: as in Fig. 3 c, but for direction probes fit on different halves of the houses: the near/far cosine stays below its ceiling at every boundary ( 0.081 vs. 0.130 at c=15 ). Bottom right: the range-split pair does not outperform a single recalibrated probe on near reads (up to 3∘ worse) and improves on it by at most 1.1∘ on far reads — a single probe suffices at all ranges.
Figure 15: Object teleportation across the workspace for distance to target object experiments. Object placements in the robot-base frame, viewed from above (robot at left, forward to the right). Gray points are all placements in the teleport datasets (906 houses). For one example house, the black point is the object’s natural placement and the blue points its 15 teleported placements, sampled uniformly over the supporting surface within the benchmark placement envelope; dashed arrows connect the natural placement to each teleport. Only the object is moved; the arm’s initial pose is unchanged.
Figure 16: Contact is decodable from the wrist-camera patches at the first layer and from the exo-camera patches and text tokens at a similar level only from the middle layers on (VLM). (Left) Held-out AUC of the contact discriminant at each of 10 layers, fit separately on five token groups: wrist-camera patches, exo-camera patches, text tokens, the final token, and the whole-sequence mean; reference lines mark a phase-timer baseline, a gripper-closure baseline, and the action expert. (Right) Per-patch AUC at layer 24 for the two cameras: the signal is spread over the whole image rather than confined to the fingers.
Figure 17: MolmoBot’s task phase subspace in the VLM backbone (vision tokens, Layer 31). (Left) LDA projection of natural rollouts, colored by phase (0.86 balanced accuracy across held-out houses; AE: 0.92). (Middle) The full three-dimensional LDA subspace, with a held-out episode’s trajectory traversing the four phase regions in order. (Right) Freezing shifts the VLM’s decoded phase toward grasp, as in the action expert (Figure 5 ): frozen reach and grasp landmarks are mostly decoded as grasp, and frozen transport and place landmarks are most often still decoded as transport and place.
Figure 18: Under the frozen read, the decoded phase shifts toward grasp in all three streams, while place landmarks are most often still decoded as place. Left to right: action expert, VLM vision tokens, VLM language tokens. Each row gives the percentage of one phase’s landmarks decoded as each phase when the policy is shown the current frame twice (bold: the true phase); the strip beneath each matrix gives the percentage decoded correctly with the usual two frames, at the same moments.
Figure 19: Per-step time course of the freezing effect on gripper commands. Fraction of chunks commanding a closed gripper at each step of the executed action chunk (shaded: 95% CI), natural vs. frozen, per phase.
Figure 20: Wrist roll sweep (Figure 6 ). Exo (third-person) camera images at several points of the sweep through the wrist’s full roll range ( −172.8∘ to +172.8∘ ), with all other joints fixed. Borders are colored by wrist angle θ .
Figure 21: The VLM maintains only a weak, pixel-derived trace of wrist orientation. A roll sweep from the same starting pose as Figure 6 , over a 332∘ range ( −156∘ to +176∘ ), read at the VLM; the VLM receives only images and text as input. (Left) PCA over the sweep activations (layer 35). The trace wraps — −156∘ and +176∘ land close together — but does not form a clean circle. (Right) The reduced-rank regression of Figure 24 at every VLM layer, on natural rollouts re-encoded at the VLM (19,089 states, 169 houses). Decoding peaks mid-stack at R2=0.31 (layer 20), a third of the action expert’s 0.87 (dotted) on the same data.
Figure 22: After a full turn of the wrist joint the action expert’s activation does not return to its start; the VLM’s returns exactly. From one pose of the roll sweep (App. C.8 ; a single house), the wrist joint (Franka joint 7) is driven through successive full turns by writing the joint value directly into the simulator, far past the joint’s physical range (shaded band), so the sweep tests the models’ inputs rather than a reachable motion. At every full turn (dashed lines, large markers) the rendered frames and text tokens are identical to the start and only the raw joint value differs, and only the action expert receives that value. (Left) Action expert, layer 9, first action token: Euclidean distance of the activation at each step from the start activation; after one turn it is 28.8 units away and does not return over the following turns. (Right) VLM, layer 20, mean over image tokens: the same distance, which varies within each turn and is exactly 0 at every full turn; the two panels are in different activation units.
Figure 23: Even the most circular reachable wrist rotation does not close. Instead of rolling about the gripper’s approach axis, we tilt the gripper 30∘ off that axis and orbit the tilt about the vertical. If driven through a full turn, this precession returns the wrist orientation exactly to its start (a closed circle in SO(3) ). The gripper position is held fixed within 5 cm while the whole arm reconfigures, and we record activations at each step. The arm’s joint limits (clipped to the 1st to 99th training percentile) stop the orbit partway: sweeping both directions from the start covers −70∘ to +158∘ ( 228∘ of the turn; angles are the realized rotation about the vertical axis). (Top) PCA over the sweep activations. Precession remains an open arc, with PC1 carrying 69% of the variance and PC2 25% . (Bottom) The exo camera image at different rotation angles along the sweep.
Figure 24: Full 3-DOF wrist orientation is decodable from natural rollouts at low rank (AE). Unlike Fig. 6 , which rotates a single roll joint and reads the code with PCA, here “wrist orientation” means the complete 3-DOF orientation of the hand, represented by the first two columns of its rotation matrix (a 6-D target). We fit reduced-rank ridge regressions of rank 1 to 6 from the activations to this target on natural rollouts (6,555 queries, 191 houses) and report R2 on held-out houses at six action-expert layers. Rank 2 recovers most of the decodable variance and rank 3 nearly all of it; higher ranks add little. Decodability peaks at layer 9.
Figure 25: Cumulative variance of gripper steering vectors in layer 32. Cumulative explained variance from PCA on the pullback steering vectors δ (App. C.9 ) as a function of the number of principal components. The first component captures ∼70% of the variance, complementing the per-component view in Figure 7 .
Figure 26: Gripper control for action expert layer 9 (Top left) A linear probe decodes the commanded gripper aperture; one principal component of the pullback steering vectors decodes it nearly as well as the full activation. (Top right) Natural-rollout activations colored by gripper state; notably, the steering vectors from each cluster do not point toward the other, unlike layer 32 in Figure 7 . (Bottom left) One component captures ∼55% of steering-vector variance. (Bottom right) One component flips the gripper but leaks into the arm joints relative to using the full 768-dimensional steering vector; more components remove this excess leakage.
Figure 27: In the VLM backbone, gripper steerability falls to the control floor by layer 28 although aperture readability stays high. The gripper pullback of App. C.9 with the edit δ added to every sequence token after VLM block L , at screen scale (32 edits per layer and objective, one query per landmark). (Left) Fraction of edits that flip the commanded gripper under the arm-hold objective of App. C.9 and under the gripper-only objective (the arm-hold and smoothness terms dropped), counted on the 29 edits whose unedited chunk has the opposite gripper sign; the gray band spans the flip rates of a control edit (the mean of the action-expert catalog, App. C.9 ). (Middle) Held-out R2 of a ridge probe from the pre-edit activation to the commanded aperture, best of the token pools at each layer; readability rises to 0.86 by layer 20. (Right) Absolute cosine between the first principal component of the arm-hold edits, a sample-limited estimate from the small per-layer catalog, and the close–open flip axis (dashed: the action-expert catalog of Figure 26 ); at the shallowest flipping layers (4 and 12) the leading component is only partly aligned with the flip axis.
Figure 28: The gripper steering vectors stay dominated by one component and aligned with the flip axis at every action-expert layer. The pullback catalog of App. C.9 , re-fit at layers 9, 18, 27, and 32. (Left) Share of steering-vector variance on the first principal component. (Middle) Stable rank of the steering vectors. (Right) Cosine between the first principal component and the flip axis (mean close edit minus mean open edit), and held-out R2 of the aperture probe restricted to that component; dashed: a random direction.
Figure 29: DROID real world behavioral manifolds. Here we show the behavioral manifolds realized by MolmoAct2 and π0.5 . The EE position and wrist orientation point clouds show the activations of all 3,321 queries, sampled from MolmoAct2 AE L16 of 36 and π0.5 VLM L7 of 18, demonstrating the structures appear across multiple streams. We plot a natural episode trajectory in the EE position subplots and the wrist roll sweep of Figure 30 in the Wrist Orientation subplots. In the Gripper plot we project the 379 landmark activations from each model’s final action-expert block onto the principal components of the pullback steering vectors, with binned steering vectors as arrows.
Figure 30: Wrist orientation PCA across models and streams. Following Fig. 6 , we record a real-robot episode in which the wrist roll joint is swept from −79∘ to +84∘ while the other joints and the scene are held approximately fixed, and plot the top two principal components of the activations for MolmoAct2 and π0.5 in both the VLM and the action expert (color: roll angle). As in MolmoBot, the sweep traces an open, semicircular arc. Because real-world data are noisier, the trajectories are considerably less smooth.
Figure 31: MolmoAct2 and π0.5 encode distance to target object in 2 orthogonal subspaces. We find that similar to MolmoBot, as shown in Fig. 3 the tested DROID model also encode distance to target object with a multi-scale ruler. We sample 26,675 timesteps from 1,998 pick and place episodes. The distance is how far the gripper is from its initial grasp location. Activations are sampled from MolmoAct2 VLM L25 of 36 and π0.5 VLM L14 of 18.
Figure 32: Task phase subspace in π0.5 and MolmoAct2. (Left) Example trajectory for a simple pick-and-place task, “Pick up the white and green brush from the table and hang it on the wall,” along with the predicted task phases. The ground-truth bar is derived from the measured gripper aperture alone, using a simple heuristic: reach until the gripper starts to close, grasp while it closes, transport while it stays closed, place after it opens. A linear classifier is trained on the top 48 principal components of activations collected across 2,000 episodes. (Right) The recovered task subspace in both models, colored by phase. Activations are taken from the π0.5 action expert at layer 9 of 18 and the MolmoAct2 action expert at layer 22 of 36.
Figure 33: Gripper steering vectors on real DROID episodes . (Top left) A linear probe decodes each model’s own gripper command from the activations projected onto the top-k principal components of the steering vectors. Dashed lines: each model’s full-activation ceiling. (Top right) Activations colored by the commanded gripper (red open, blue closed); the steering vectors point from each cluster toward the other. (Bottom left) One component captures 59 % of π0.5 ’s steering-vector variance (stable rank 1.69) but only 31% of MolmoAct2’s (stable rank 3.21).(Bottom right) One component flips the gripper (94–96% of attempts, both models) but leaks into the arm joints at 2.4–4.3× the full- δ optimum. Activations are recorded and steered at the final action-expert block (MolmoAct2 layer 35 of 36, π0.5 layer 17 of 18, counting from 0).
Figure 34: The seven task variables. An overview of each variable we study.
Figure 35: Arm and object placements in the crossed teleport harvest used for EE and target position. Robot-base frame viewed from above (left) and from the side, with height above the object’s resting position (right). Gray points are arm poses and violet diamonds object placements across all houses; the dashed outline is the box arm poses are sampled in. For one example house, the black point is the home pose, blue points its ten arm poses, and violet diamonds its five object placements; unlike Figure 15 , both the arm and the object move.
Figure 36: The VLM keeps the target’s position but no map of the workspace; other objects are encoded per image patch. (Left) A table-occupancy map of the target, read from VLM activations (last token, layer 21), is no more accurate than the same readout applied to the target’s three true coordinates (dashed). (Middle) One readout shared across all image-patch tokens detects whether an object the instruction does not name is visible in that patch, well above patch-position and patch-color baselines (exo and wrist cameras, layers 8 and 20). (Right) With scene means removed, a readout of table position fit on all visible objects does not track the target as it moves, whereas one fit on the instructed target alone does on the exo camera.
At its core, robotic manipulation is a problem of vision-to-geometry mapping (f(v)→G). Physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional VLA and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce Progressive Volumetric Modulation and jointly train action and 3D property prediction to preserve geometric representations. Extensive experiments validate the effectiveness of our approach. Across simulation benchmarks, VGA outperforms leading VLA, 3D-VLA, and WAM baselines, including π0.5, OpenVLA-OFT, GeoVLA, and Motus. In real-world deployments, VGA surpasses π0.5 under unseen viewpoints and accurately follows language instructions for target grasping. These results highlight that operating on native 3D representations, rather than relying primarily on language or video priors, offers a promising direction toward generalizable physical intelligence. Project page: https://hcplab-sysu.github.io/VisionGeometryActionModel.
Zijian Song, Qichang Li, Jiawei Zhou +4
Sun Yat-sen University · LongCat, Meituan · X-Era AI Lab +2
Vision-language-action (VLA) models have rapidly advanced generalist robot manipulation by harnessing the semantic knowledge of pretrained vision-language backbones. However, their visual representations are typically grounded in 2D image coordinates rather than calibrated camera geometry. This limitation is especially pronounced in multi-camera systems, where views are coupled by known intrinsics and extrinsics but processed as independent images. We propose G3VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA without altering its action space or original action-learning objective. G3VLA combines intrinsic-conditioned ray embeddings, projective positional encoding (PRoPE), and bidirectional cross-view fusion to establish geometrically consistent representations across camera views. Geometric supervision is provided by ground-truth point maps or confidence-gated π3X teacher predictions, enabling training without depth sensors or manual geometric annotations. On π0, G3VLA consistently improves performance on LIBERO and RoboCasa24 under both supervision regimes, while ground-truth supervision improves five-task RoboTwin2.0 averages with three cameras. Experiments on camera-shifted LIBERO-Goal and real robots demonstrate improved robustness to unseen camera viewpoints. Controlled calibration studies across π0 and π0.5 show that these gains depend on correct camera geometry while remaining stable under mild calibration perturbations. These results demonstrate that grounding VLA visual tokens in calibrated camera geometry provides a simple and general mechanism for improving geometric consistency, multi-view reasoning, and viewpoint robustness in pretrained robot policies.
Yue Peng, Yongzhe Zhao, Artur Habuda +5
New York University Shanghai · Technical University of Denmark · MBZUAI - Mohamed bin Zayed University of Artificial Intelligence
Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observations, while geometry-aware approaches often require additional 3D information. However, geometry inferred independently from individual observations is often discarded rather than accumulated into a persistent task representation. Our key insight is that action generation needs a compact, task-relevant geometric abstraction that can be continuously updated, rather than a complete, renderable 3D reconstruction at every step. We introduce GeoMem-VLA, a VLA framework that internalizes explicit 3D supervision into dedicated geometric tokens and maintains them as temporal memory for action generation, without directly aligning the VLM's native visual embeddings with the teacher feature space. During training, a frozen geometry foundation model supervises learnable GeoPlan tokens through a geometric flow-matching objective over the current scene point map and future end-effector tracks. These tokens are integrated with the memory of observation representations, where historical information is retrieved, adaptively fused with current observations, and consolidated to retain event frames as the task progresses. At inference, the policy retrieves historical representations and fuses them with the current working representation to predict actions, without invoking the 3D teacher or explicitly reconstructing the scene. After benchmark-specific downstream adaptation, GeoMem-VLA achieves 98.9% average success on LIBERO and 91.10% / 91.24% on RoboTwin's clean/randomized settings. Across five real-world tasks, GeoMem-VLA achieves 78.0% average success. These results suggest that internalizing and remembering 3D scene understanding is a promising approach to robot control.