ExecVLA: Following Fine-Grained Execution Constraints in Vision-Language-Action Models with Bi-Level Action Representation

Gaoge Han; Zhengqing Gao; Ziwen Li; Jiaxin Huang; Shaoli Huang; Fakhri Karray; Mingming Gong; Tongliang Liu

cs.ROMar 18, 2026

ExecVLA: Following Fine-Grained Execution Constraints in Vision-Language-Action Models with Bi-Level Action Representation

Authors: Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu

Organizations: MBZUAI.

Abstract

We study fine-grained execution-constraint following in vision-language-action (VLA) models. Given an invariant task goal, the policy must follow instruction-specified execution constraints, including interaction targets, motion patterns, spatial relations, and terminal configurations. This setting exposes a limitation of goal-oriented VLAs: trajectories that complete the same task are not interchangeable when the instruction specifies how the task must be executed. We propose ExecVLA, a framework that separates a goal-oriented component from an execution-specific component through a bi-level action representation and supervised bi-level reasoning tokens. We further introduce explicit goal-invariance and execution-predictability objectives so that the goal-level representation remains stable across executions of the same goal, while the execution-level representation retains the constraints that distinguish those executions. We construct execution-constraint-following datasets in simulation and on a Realman-75 robot, with goal and fine-grained reasoning annotations. Experiments on LIBERO and the real robot show improved goal completion and, more importantly, substantially more reliable adherence to instruction-specified execution constraints.

Figures & tables

Figure 1 : Comparison of Vanilla VLAs vs. KineVLA. Vanilla VLAs ( Kim et al., 2025 ; Wang et al., 2025 ) , which accept coarse command inputs (e.g., “place the wine bottle on the drawer”) and produce relatively fixed bottle label orientation actions. In contrast, Our KineVLA can processes fine kinematic commands (e.g., “control the wine bottle to face a specific orientation on the cabinet”) and generate robot end-effector actions oriented front, back, left, or right.

Figure 2 : A task example from our proposed Kinematics-Rich Datasets : In contrast to coarse task instructions that aim to complete a general goal, our approach captures diverse, fine-grained kinematic variations for an action instruction and their temporal evolution across multiple key action stages. These variations encompass {A} Object Part, {B} Action Constraint, and {C} Target Relations, with corresponding images of key actions displayed from left to right. Our KineVLA method is designed to address this challenge, excelling at perceiving these multi-faceted details to achieve precise manipulation.

Figure 3 : Overview of our framework. Our approach decouples low-frequency, goal-level control from fine-grained kinematic refinements to effectively handle kinematics-rich tasks. (a) The proposed Bi-Level RVQ-VAE learns hierarchical action representations (Sec. 3.1 ), while (b) the KineVLA framework addresses kinematics-rich tasks through bi-level generation (Sec. 3.2 ).

Figure 4 : Experimental Results and Comparisons. We benchmark our method across the three proposed kinematics-aware datasets, encompassing both simulation and real-world robotic experiments. The left two figures illustrate example environments, while the bar chart on the right presents the goal and kinematic success rates.

Figure 5 : Task execution examples using KineVLA. From left to right, the figure shows the kinematics-rich instruction input, the initial state of the environment, followed by the bi-level reasoning text and action tokens generated by the KineVLA model. Next are two columns illustrating the key robot action states, and finally, the resulting final state after the robot executes the task.

Dataset	BLEU	BERTScore	Intervention (%)
LIBERO-Goal-Relabled	0.93	0.94	76.5 $\rightarrow$ 52.4
Kine-LIBERO	0.95	0.96	70.4 $\rightarrow$ 48.6
Kine-Realman	0.92	0.94	65.0 $\rightarrow$ 42.4

Table 1 : Bi-level reasoning evaluation using the BLEU, BERTScore and Intervention metrics.

Methods	Inference Time
OpenVLA	237ms
$\pi_{0.5}$	86ms
VQ-VLA	84ms
KineVLA	96ms

Table 2 : Inference Speed Comparison.

Dataset	Baseline	+ Bi-Rep	+ Bi-Rea	+ MI
LIBERO-Goal-Relabled	70.0	73.4	75.5	76.5
Kine-LIBERO	61.5	66.8	68.6	70.4

Table 3 : Ablation studies of KineVLA components. Success rates (%) across two kinematics-rich datasets on LIBERO platform. “Baseline” refers to the standard vector-quantized action tokenization using OpenVLA. “+ Bi-Rep” denotes the use of our Bi-Level action representation. “+ Bi-Rea” indicates the incorporation of bi-level reasoning token. “+ MI” refers to the addition of mutual information optimization between the reasoning text and the action.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Setting	Frames	Language	Reasoning Text	Vision	Action Space
LIBERO-Goal-Relabeled	Simulation	63K	Kinematics-rich	Bi-Level	Front, Wrist	EEF Pose
Kine-LIBERO	Simulation	266K	Kinematics-rich	Bi-Level	Front, Wrist	EEF Pose
Kine-Realman-75	Real robot	49K	Kinematics-rich	Bi-Level	Front, Wrist	Joint Positions

Appendix

Table 4: Summary of our proposed datasets collected via SpaceMouse and puppet manipulation. In each task, we gather 50 demonstrations.

Figure 6 : Execution results on the Kine-LIBERO dataset. The serial number corresponds to the entry in Table 6 .

Figure 7 : Execution results on the Kine-Realman-75 dataset. The serial number corresponds to the entry in Table 7 .

No.	Kinematic language instruction	Coarse-grained CoT	Fine-grained CoT
1	grasp the left edge of the bowl, move it forward, and place it precisely at the center of the plate	put the bowl on the plate	1. Robot Action-level Constraint: Trajectory shape - forward linear movement; Contact timing - grasp then place; Ordinal relations - sequential grasp, move, place. 2. Object-level Targeting: Target - bowl’s left edge for grasp, plate’s center for placement; Pose/Facing/Angle - precise placement with implied alignment.
2	grab the wine bottle by the neck and place it on the rack	put the wine bottle on the rack	1. Robot Action-level Constraint: Contact Timing - Must initiate/break contact (grab/release). Trajectory Shape - Requires transfer motion from source (neck) to target (rack). Ordinal Relations - Strict sequence: grab before place. 2. Object-level Targeting: Grasp Target - The neck (specific part) of wine bottle. Place Target - The rack (specific object); final pose constrained by rack structure (must be placed on it).
3	press the handle to open the top drawer to its maximum extension, then grasp the left edge of the bowl and place it in the center of the drawer	open the top drawer and put the bowl inside	1. Robot Action-level Constraint: State/Trajectory: Press, then grasp and place. Contact Timing: Press handle first, then grasp bowl after drawer is open. Ordinal Relations: Sequential: open drawer, then grasp bowl, then place bowl. 2. Object-level Targeting: Handle/Surface/Edge: Handle (of drawer), left edge (of bowl). Pose/Facing/Angle: Maximum extension (drawer state), center (target placement pose).
4	grab the cream cheese by the middle and place it in the center of the bowl	put the cream cheese in the bowl	1. Robot Action-level Constraint: Trajectory shape - Straight-line placement motion; Contact timing - Grasp first, then place; Ordinal relations - Sequential (grab, then place). 2.Object-level Targeting: Handle/Surface - Target middle of object (cream cheese); Pose/Facing - Orientation-agnostic grasp; Placement - Center of bowl (specific positional target).
5	hold the wine bottle firmly by the neck and position it on the upper section of the cabinet top, orienting the bottle so that its label is facing to the left	put the wine bottle on top of the cabinet	1. Robot Action-level Constraint: State/Trajectory: Hold firmly, position on upper section. Contact Timing: Grip neck before positioning. Ordinal Relations: Orient after positioning. 2. Object-level Targeting: Handle/Edge/Surface: Neck (grip), cabinet top (placement). Pose/Facing/Angle: Label facing left.
6	place the end-effector at the center of the plate and move it leftward then backward along the table plane to reposition the plate about close in front of the stove	push the plate to the front of the stove	1. Robot Action-level Constraint: State/Trajectory: Move end-effector leftward then backward along the table plane. Contact Timing: Maintain contact with the plate. Ordinal Relations: Sequential motion: first leftward, then backward. 2. Object-level Targeting: Target Part: Center of the plate (initial contact). Pose/Facing/Angle: Align end-effector to plate center; movement constrained to table plane. Final Placement: close in front of the stove.

Appendix

Table 5: LIBERO-Goal-Relabeled Dataset.

No.	Kinematic language instruction	Coarse-grained CoT	Fine-grained CoT
1	grab the right edge of this bowl and move it along a diagonal path to the upper section of the cabinet	put the bowl on top of the cabinet	1. Robot Action-level Constraint: Trajectory shape - Diagonal path; Contact timing - After grabbing, then move 2. Object-level Targeting: Target component - Right edge of the bowl; Spatial relation - Upper section of the cabinet
2	grab the left edge of this bowl and move it along a diagonal path to the upper section of the cabinet	put the bowl on top of the cabinet	1. Robot Action-level Constraint: Trajectory shape - Diagonal path; Contact timing - After grabbing, then move 2. Object-level Targeting: Target component - Left edge of the bowl; Spatial relation - Upper section of the cabinet
3	grab the right edge of this bowl and move it along a diagonal path to the bottom section of the cabinet	put the bowl on top of the cabinet	1. Robot Action-level Constraint: Trajectory shape - Diagonal path; Contact timing - After grabbing, then move 2. Object-level Targeting: Target component - Right edge of the bowl; Spatial relation - Bottom section of the cabinet
4	grab the left edge of this bowl and move it along a diagonal path to the bottom section of the cabinet	put the bowl on top of the cabinet	1. Robot Action-level Constraint: Trajectory shape - Diagonal path; Contact timing - After grabbing, then move 2. Object-level Targeting: Target component - Left edge of the bowl; Spatial relation - Bottom section of the cabinet
5	hold the handle of the middle cabinet drawer and pull it open completely	open the middle drawer of the cabinet	1. Robot Action-level Constraint: Contact Timing: Initiate contact Trajectory Shape: Linear pulling. Ordinal Relations: Grasp then pull 2. Object-level Targeting: Target Part: Middle cabinet drawer handle. Action on Target: Pull open completely
6	vertically hold the handle of the middle cabinet drawer and pull it open completely	open the middle drawer of the cabinet	1. Robot Action-level Constraint: Contact Timing: Initiate contact. Trajectory Shape: Linear pulling. Ordinal Relations: Grasp then pull. 2. Object-level Targeting: Target Part: Middle cabinet drawer handle. Action on Target: Pull open completely

Appendix

Table 6 : Kine-LIBERO Dataset

No.	Kinematic language instruction	Coarse-grained CoT	Fine-grained CoT
1	pick up the bread and move it to the left front	move the bread	Robot Action: Trajectory Shape - To the left front; Ordinal Relations - Pick first, then move. Object Targeting: Target Object - The bread; Spatial Relation - Move it to a location (left front).
2	pick up the bread and move it to the right front	move the bread	Robot Action: Trajectory Shape - To the right front; Ordinal Relations - Pick first, then move. Object Targeting: Target Object - The bread; Spatial Relation - Move it to a location (right front).
3	pick up the bread and move it to the front	move the bread	Robot Action: Trajectory Shape - To the front; Ordinal Relations - Pick first, then move. Object Targeting: Target Object - The bread; Spatial Relation - Move it to a location (front).
4	hold the carrot by the upper part and place it on the right side of the plate	place the carrot in the plate	Robot Action-level Constraint: State/Trajectory - Move carrot from held to placement pose. Ordinal Relations - Grasp first, then place. Object-level Targeting: Carrot - Target handle (upper part). Plate - Target surface (top side). Spatial Relation - Final pose on the right side of the plate.
5	hold the carrot by the upper part and place it on the left side of the plate	place the carrot in the plate	Robot Action-level Constraint: State/Trajectory - Move carrot from held to placement pose. Ordinal Relations - Grasp first, then place. Object-level Targeting: Carrot - Target handle (upper part). Plate - Target surface (top side). Spatial Relation - Final pose on the left side of the plate.
6	hold the carrot by the lower part and place it on the left side of the plate	place the carrot in the plate	Robot Action-level Constraint: State/Trajectory - Move carrot from held to placement pose. Ordinal Relations - Grasp first, then place. Object-level Targeting: Carrot - Target handle (lower part). Plate - Target surface (top side). Spatial Relation - Final pose on the left side of the plate.

Appendix

Table 7 : Kine-Realman Dataset

Explore similar work

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies
May 26, 2026Xintong Hu, Xuhong Huang, Jinyu Zhang +11Language-Conditioned Robot Manipulation Vision-Language-Action Models
XLANG Lab, The University of Hong Kong · Qwen Team, Alibaba Inc.
Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Dynamic Execution Commitment of Vision-Language-Action Models
May 12, 2026Feng Chen, Xianghui Wang, Yuxuan Chen +4VLM Robustness Efficient VLM Inference
University of Adelaide · Sichuan University · Shanghai Jiao Tong University +3

May 26, 2026, cs.ROJ/K move · Enter open · S save

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao+10

XLANG Lab, The University of Hong Kong · Qwen Team, Alibaba Inc.

Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Language-Conditioned Robot Control

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 11,631 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/

Abstract

Explore similar work

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Dynamic Execution Commitment of Vision-Language-Action Models