Task π 0 \pi_{0} π 0 π 0.5 \pi_{0.5} π 0.5 Baseline Selection. To comprehensively evaluate our approach, we benchmark HiVLA against four state-of-the-art (SOTA) models: π 0 \pi_{0} π 0 π 0.5 \pi_{0.5} π 0.5 π 0 \pi_{0} π 0 π 0.5 \pi_{0.5} π 0.5 We note that open-source, visual-grounded hierarchical systems designed for general manipulation remain scarce, which informs our baseline choices. For instance, DexGraspVLA [ 42 ] is strictly restricted to multi-object grasping and lacks cross-task generalization, precluding a feasible comparison. Proprietary systems like Gemini Robotics [ 36 ] and HiRobot [ 31 ] are closed-source. Furthermore, InterleaveVLA [ 13 ] focuses primarily on object-centric policies rather than offering a complete hierarchical system.
System Latency Analysis. Our decoupled architecture resolves the inherent frequency mismatch between slow vision-language reasoning and high-speed motor control via asynchronous inference. While the unoptimized VLM Planner requires 1.9s per reasoning step, leaving ample room for software acceleration, the DiT Action Policy efficiently infers a 16-step action chunk in merely 0.162s. By executing the semantic planner in parallel with the fast control policy and maintaining temporal consistency through mechanisms like real-time tracking, the system effectively bridges this latency gap to achieve an 8 Hz 8\text{Hz} 8 Hz 5.2 Evaluation in RoboTwin Platform
Evaluation Tasks and Protocols. For a comprehensive assessment, we benchmark all models across a suite of 9 tasks, categorized into four Easy Tasks and five Hard Tasks . Easy tasks typically require a single skill and evaluate precise visual perception ( e.g. , grasping small objects like a bell or a stapler). Hard tasks involve sequences of multiple skills or demand advanced spatial and semantic reasoning. For instance, ‘Stack 3 Blocks’ requires the model to sequentially infer the correct colored block to manipulate based on a specific visual order, while ‘Click 3 Bells’ presents three identical bells, forcing the model to rely strictly on spatial language reasoning (‘left’, ‘center’, ‘right’) to disambiguate the target. Visualizations of these selected tasks are provided in Fig. 3 , with a comprehensive list of textual instructions and visual states detailed in the Appendix. For each task, we conduct 100 independent trials under unseen environment configurations, reporting the average success rate over the last three saved checkpoints to ensure statistical stability.
VLM Planner Training and Validation. To validate the high-level reasoning capabilities of our VLM Planner, we curated a specialized 210K-instance dialogue dataset derived from HiVLA-HD. Fine-tuning the Qwen3-VL [ 1 ] 8B model on this domain-specific data yields highly robust semantic planning, achieving a bounding box grounding accuracy (mIoU) of 90.37% and a strict exact-match sub-task prediction of 98.57% (with comprehensive evaluations detailed in the Appendix). Crucially, our decoupled architecture makes the VLM planner readily replaceable. This flexibility allows the system to either undergo lightweight fine-tuning for specific operational domains, or directly integrate off-the-shelf VLMs for zero-shot deployment. While we adopt fine-tuned 8B model for all subsequent evaluations to maximize task performance, this extensible design easily accommodates future advancements in more powerful VLM agents.
Action Policy Training and Main Results. To ensure a fair comparison, all baselines are all fine-tuned on the HiVLA-HD dataset for 150K steps using two H200 GPUs (batch size 64). Notably, HiVLA’s DiT is initialized from H-RDT weights pre-trained on the EgoDex dataset, where we directly copy the weights from the global image cross-attention layer to initialize our novel local image cross-attention layer.
As presented in Sec. 5.1 , HiVLA achieves an unparalleled total average success rate of 83.3%, significantly outperforming SOTA coupled VLAs ( π 0 \pi_{0} π 0 π 0.5 \pi_{0.5} π 0.5 < 40 % <40\% < 40% Corrupted Modality Error Injection Rate 0% 20% 40% 60% 80% 100% Noise on BBox 83.3% 78.5% 74.0% 70.8% 59.8% 57.0% Noise on Task (Lang) 83.3% 69.3% 49.3% 36.0% 24.3% 12.0% Noise on Both 83.3% 62.0% 42.5% 33.3% 23.0% 17.3% Table 2 : Robustness evaluation of the Action Expert against guidance perturbations. The policy is highly resilient to Bbox noise but strictly adheres to language instructions.
Skill Decomposition and Error Correction. Our ablation variant, Ours (w/o Skill), further isolates the impact of granular language instructions. While performance is comparable on Easy tasks (where the global instruction intrinsically matches a single skill), replacing specific sub-task skills with the global instruction on Hard tasks causes an 8.8% performance drop. This confirms that decomposed, “one-to-one" language conditions drastically reduce the cognitive load on the diffusion policy, allowing it to focus purely on local geometry and execution. Furthermore, we observed a compelling emergent error-correction property: if the DiT policy fails a grasp (a “phantom execution"), the VLM Planner acts as an independent semantic supervisor. Recognizing the sub-task as incomplete, it seamlessly re-issues the visual-language command, enabling the system to re-attempt the skill—a resilience unattainable in standard coupled VLAs.
Robustness to Planner Errors. A pervasive critique of hierarchical systems is their susceptibility to compounding errors, where a planner’s mistake irreversibly crashes the downstream policy. To address this, we subjected our Action Expert to rigorous perturbation testing ( Tab. 2 ) by injecting calibrated noise into the inferred bounding boxes and language instructions. The results reveal a highly desirable decoupling: the policy demonstrates strong resilience to spatial noise. Even with 100% shifting in the target bounding box, it retains a 57.0% success rate, effectively leveraging the auxiliary global image features to self-correct and localize the true target. Conversely, injecting noise into the language instructions causes a proportional degradation in performance, precisely matching the error injection rate. This confirms the policy’s strict semantic compliance, proving that our architecture successfully balances robust visual adaptability with absolute adherence to language commands.
5.3 Evaluation in the Real World
To further validate real-world applicability and robustness of our visual-grounded hierarchical design, we deployed HiVLA in a physical robotic environment.
Real-World Task Setup and Protocol. As visualized in Fig. 3 , our experiments span 7 object categories and encompass 16 distinct sub-type scenarios. Rather than evaluating on conventional easy tasks, we specifically designed these scenarios to stress-test strong cross-environmental generalization and precise instruction following. By utilizing complex combinations of objects with varying colors and spatial arrangements, the tasks range from manipulating a single primitive to selecting a specific target ( e.g . , a red block or a green cup) from dense, multi-object clutter.
Method Click Bell Pick & Place Cup Pick & Place Block 1 Bell 2 Bells 1 Cup 3 Cups 1 Block 3 Blocks H-RDT 8/30 9/30 4/30 0/30 9/30 0/30 Ours 13/30 17/30 21/30 6/30 20/30 7/30 Table 3 : Real-world success rates. HiVLA excels in multi-object cluttered scenarios requiring semantic grounding, where baseline models struggle.
For model training, we collected a dataset of 360 teleoperated episodes, automatically annotated with precise bounding boxes via GroundingDINO [ 24 ] and SAM2 [ 28 ] . Both H-RDT and HiVLA were initialized from their simulation-trained checkpoints to leverage structural priors, and subsequently fine-tuned for 80K steps on real-world data. During the evaluation phase, each task was attempted for 30 trials, with object positions randomized to prevent rote memorization and rigorously assess policy robustness.
Success Rate Analysis. The real-world success rates are detailed in Tab. 3 . It is crucial to note that our evaluation suite intentionally targets hard, strong-generalization scenarios requiring rigorous semantic reasoning. Consequently, the baseline H-RDT exhibits performance degradation. While it performs adequately in isolated single-object scenarios, its success rate collapses to nearly zero in multi-object clutter ( e.g. , ‘3 Cups’, ‘3 Blocks’). Relying solely on global visual features, H-RDT lacks the fine-grained grounding required to disambiguate identical shapes using color attributes or spatial commands. In contrast, HiVLA’s hierarchical decoupling relieves the DiT Action Expert of the global reasoning burden, allowing it to efficiently map the sparse, limited real-world data to precise local visual-language conditions. As a result, HiVLA effectively navigates complex, cluttered scenes, executing sub-skills with remarkable accuracy and generalizing robustly across demanding physical tasks.
5.4 Ablation Study
We conduct ablation studies on the DiT Action Expert’s architecture to analyze the efficacy of visual representations and cross-attention guidance strategies.
Guidance Injection Strategy. The order in which conditions are injected into the DiT via cross-attention heavily impacts policy learning. We expanded our ablation to isolate guidance contributions ( Sec. 5.4 ). Relying solely on Local or Global visual features yields suboptimal results ( ∼ \sim ∼ → \to → → \to → 83.3 % 83.3\% 83.3% Table 4 : Ablation study on guidance injection strategies and visual-grounding components. Best results are bold .