Vision-language-action(VLA) models have shown great promise as generalist policies for a large range of relatively simple tasks. However, they demonstrate limited performance on more complex tasks, such as those requiring complex spatial or semantic understanding, manipulation in clutter, or precise manipulation. We propose OMNIGUIDE, a flexible framework that improves VLA performance on such tasks by leveraging arbitrary sources of guidance, such as 3D foundation models, semantic-reasoning VLMs, and human pose models. We show how many kinds of guidance can be naturally expressed as differentiable energy functions with task-specific attractors and repellers located in 3D space, that influence the sampling of VLA actions. In this way, OMNIGUIDE enables guidance sources with complementary task-relevant strengths to improve a VLA model's performance on challenging tasks. Extensive experiments in both simulation and real-world environments, across diverse sources of guidance, demonstrate that OMNIGUIDE enhances the performance of state-of-the-art generalist policies (e.g., π0.5, GR00T N1.6) significantly across success and safety rates. Critically, our unified framework matches or surpasses the performance of prior methods designed to incorporate specific sources of guidance into VLA policies. Project Page: this ; url
Figures & tables
Fig. 1: OmniGuide unifies different kinds of guidance via attractive and repulsive fields to improve the performance of generalist robot policies.
Fig. 2: Method Overview. For each denoising step, OmniGuide first estimates the clean action chunk A~τ by the base policy vθ , and then decodes it into joint space. A differentiable dynamics/kinematic model is then used to obtain the robot’s Cartesian trajectories X , which are evaluated using the energy functions Ly extracted from foundation models. Finally, the gradient will be backpropagated through the robot model and all neural networks, yielding a guidance vector on the noisy latent action chunk Aτ .
Fig. 3: Guidance Visualization . We visualize the guidance gradient(lines) on the predicted Cartesian trajectories(dots), which will be backpropagated to the latent space as denoising guidance. The guidance gradient from collision energy repels the trajectories from obstacles, and the semantic energy gradient attracts the gripper for the grounded target. The two guidances are naturally blended in the space, yielding a joint guidance gradient which steers the denoising to a safe and task-oriented state.
Fig. 4: Qualitative Results of Simulation Experiments. OmniGuide demonstrates strong generalization capabilities and flexibility across diverse tasks. We show the individual and joint effects of our method on different tasks. Left: collision avoidance guidance for the TurnSinkSpout task. Middle: semantic grounding guidance for the Multi-Choice task. Right: combination of the two guidance for the Multi-Choice in a clutter scene.
Fig. 5: Quantitative Results of Simulation Experiments. OmniGuide is an effective method for enhancing base VLA’s task performance and safety. Each component, including the initialization and denoising guidance, yields an improvement over the base policy, with denoising guidance being more effective, while our combined method produces the largest boost.
Fig. 6: Real-world Qualitative Results : Our method can handle different types of guidance including 3D awareness (left), semantic reasoning (middle), and human demonstration (right). The top row shows the base π0.5 policy struggling to complete the tasks while the bottom row features the method’s successes. Arrows illustrate the end effector trajectories.
Fig. 7: Guidance Strength Sensitivity Analysis: As the collision guidance strength increases, the safety rate improves. However, excessively high guidance strength interferes with the base policy’s naturalness prior, eventually hindering task performance. Additionally, the semi-logarithmic plot reveals there is a wide range of viable hyper-parameters that can balance between the policy’s prior and the external guidance, achieving both high success and safety rates.
Fig. 8: Composing guidance fields : Semantic guidance improves the success rate while collision guidance improves the safety rate. By composing both guidance fields, OmniGuide simultaneously enhances safety and task performance.
Fig. 9: Quantitative Results of Real-world Experiments. OmniGuide consistently and significantly outperforms the base VLA and other specialized methods across three guidance modalities and nine tasks.
Fig. 10: Latency Analysis: we parallelize three expensive operations including KV cache, CLIP, and VGGT computation to improve latency. Compared to the base VLA, OmniGuide is about two times slower. Nonetheless, it is still fast enough at 15 Hz to be deployed as a reactive policy in the real-world.
Fig. 11: Hardware Setup. We use a standard 7-DOF Franka Emika Panda arm with a Robotiq 2-fingered gripper and three Zed 2 stereo cameras to provide left, right and overhead views for 3D reconstruction.
Fig. 12: VGGT 3D Reconstruction . From three camera views, we construct a 3D point cloud of the scene. We have a left, right, and overhead camera view. We also ensure that the scene has proper metric scale by comparing between the ground-truth camera pairwise distance against the VGGT’s predicted camera distance, and rescaling the depth accordingly. Shaded blue regions are the rendering of the robots in simulation by Pybullet [ 18 ] using the calibrated camera extrinsic and then super-imposed on the images . The robot is removed from the scene point-cloud.
Fig. 13: Detecting Task-relevant Regions . We use patch-level CLIP to compute 3D visual features of the point cloud. We then compute the cosine similarity of each 3D point against the language instruction. The top image is the heatmap of similarities for the task “put the banana in the purple bowl”. Task-relevant regions are defined to be the 3D points scoring in the top 95% percentile in CLIP similarity. The bottom image shows the filtered point cloud where the banana and the purple bowl were correctly removed using our procedure.
Fig. 14: VLM Pointing Guidance . (Top) First, the VLM points to all interesting points in an image. (Bottom) Then, we visually highlight each point in the image, and ask the VLM whether that point is relevant to a given task. One point is kept at the end. In this example, the VLM successfully localized the relevant object for the task of putting the object in a bowl closer to a named celebrity.
In this paper, we propose GTA-VLA(Guide, Think, Act), an interactive Vision-Language-Action (VLA) framework that enables spatially steerable embodied reasoning by allowing users to guide robot policies with explicit visual cues. Existing VLA models learn a direct "Sense-to-Act" mapping from multimodal observations to robot actions. While effective within the training distribution, such tightly coupled policies are brittle under out-of-domain (OOD) shifts and difficult to correct when failures occur. Although recent embodied Chain-of-Thought (CoT) approaches expose intermediate reasoning, they still lack a mechanism for incorporating human spatial guidance, limiting their ability to resolve visual ambiguities or recover from mistakes. To address this gap, our framework allows users to optionally guide the policy with spatial priors, such as affordance points, boxes, and traces, which the subsequent reasoning process can directly condition on. Based on these inputs, the model generates a unified spatial-visual Chain-of-Thought that integrates external guidance with internal task planning, aligning human visual intent with autonomous decision-making. For practical deployment, we further couple the reasoning module with a lightweight reactive action head for efficient action execution. Extensive experiments demonstrate the effectiveness of our approach. On the in-domain SimplerEnv WidowX benchmark, our framework achieves a state-of-the-art 81.2% success rate. Under OOD visual shifts and spatial ambiguities, a single visual interaction substantially improves task success over existing methods, highlighting the value of interactive reasoning for failure recovery in embodied control. More details of the project can be found here: https://github.com/FutianLabs/GTA-VLA.
Yiran Ling, Qing Lian, Jinghang Li +6
Faculty of Computing, Harbin Institute of Technology · International Digital Economy Academy (IDEA) · National Key Laboratory of Smart Farm Technologies and Systems +4
Vision-Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel language instructions. Off-the-shelf vision-language models (VLMs) often provide stronger generalization, but cannot directly control robot actions. To combine the common sense of VLMs with VLA control, we propose Guided Trace VLA (GT-VLA), a steerable framework that accepts guidance from an external generalist VLM through trace-conditioned action generation. GT-VLA uses a generalist model to identify semantic guidance for the current skill, converts this guidance into a 2D visual trace, and conditions its action policy on the resulting trace-rendered observation. This design separates semantic target acquisition, trace generation, and low-level action execution, allowing high-level guidance to propagate to robot actions. GT-VLA uses a Mixture-of-Experts architecture with skill-specific trace and action modules for robust execution. We evaluate GT-VLA on LIBERO and a physical robot platform, showing improved generalization over recent VLA baselines in both settings. The code and additional supplemental materials are available on our project website at https://ivaniz.github.io/gt-vla/.
Ninghan Zhong, Jing-Chen Peng, Sriram Vishwanath
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA.
While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.
Yanyan Zhang, Disheng Liu, Xinpeng Li +8
Case Western Reserve University Cleveland, OH, USA