In the practical deployment of vision--language--action policies, handling long-tailed task distributions is a common challenge due to the highly imbalanced composition of robot demonstration datasets. To investigate this issue, we study the head-task action bias in shared robot policies, particularly focusing on its relationship with task-specific visual evidence. Our experimental analysis shows that, when the visual evidence required by a tail task is insufficient, the policy increasingly falls back on action patterns reinforced by data-rich head tasks, resulting in degraded performance on under-represented tasks. Motivated by this observation, we propose Evidence-Guided Counterfactual Learning, a training framework for long-tailed robot policies. Our method identifies decision-relevant visual evidence using instruction-related and action-related signals, and constructs counterfactual observations by removing the selected evidence. The resulting counterfactual predictions are used to strengthen the association among task instructions, visual evidence, and actions, with the supervision adaptively adjusted according to the number of demonstrations available for each task. The proposed framework requires neither additional training data nor region-level annotations and incurs no additional computation during deployment. Extensive experiments across simulated benchmarks, different VLA architectures, and real-world robot platforms demonstrate that our method robustly and consistently improves long-tailed robot policy learning, particularly for tail tasks, while maintaining reliable performance on head tasks.
Figures & tables
Figure 1: Head vs.tail task performance and intermediate-level visual grounding. (a)Success rates for policies trained on FULL and LT. (b) Visual attention for “Pick up the black bowl next to the plate and place it on the plate.” (c) Visual attention for “Put the wine bottle on the rack.”
Figure 2: Diagnosing Head Task Action Bias. (a) Two-task environment. (b) Action distributions before and after relational masking at the same state under a tail-task instruction. (c) Head-Prior Score and Head-Action Rate versus task imbalance.
Figure 3: Overview of ECL. (I) Joint language–action relevance selects top- K visual tokens for masking. (II) A shared policy produces factual, masked, and contrastive training outputs, supervised by LBC , Lmask , and LCF , respectively. (III) Task frequency determines the per-sample weight of the counterfactual loss. At deployment, only the factual policy is used.
MiniVLA Success Rate (%)
π0.5 Success Rate (%, synthetic)
Task
FULL
LT(Baseline)
APA
ECL(Ours)
FULL
LT(Baseline)
APA
ECL(Ours)
Task 0
56.6 ±4.0
68.4 ±2.3
72.6 ±2.3
80.0 ±2.2
56.7 ±2.4
72.4 ±3.4
69.5 ±1.6
75.6 ±2.5
Task 1
78.2 ±4.0
49.6 ±3.6
67.3 ±1.4
68.9 ±1.3
61.5 ±3.5
64.4 ±1.6
65.8 ±1.9
71.7 ±1.8
Task 2
72.7 ±3.8
62.2 ±3.2
66.1 ±1.8
74.7 ±1.8
72.6 ±2.4
58.7 ±2.4
57.8 ±1.5
66.4 ±0.7
Task 3
50.3 ±3.8
55.2 ±3.5
70.0 ±2.1
83.5 ±1.8
50.2 ±1.6
52.6 ±2.6
55.4 ±1.9
59.6 ±0.4
Task 4
66.3 ±4.2
46.7 ±3.6
56.5 ±1.6
64.7 ±1.4
68.3 ±2.0
45.8 ±1.5
43.3 ±3.4
52.1 ±3.4
Table 1: Per-task success rates (%, mean ± binomial standard deviation of the rate) on the ten-task LIBERO-Core benchmark. The table reports four training configurations for MiniVLA and π0.5 .The Average standard deviation is computed across the three seed-wise mean success rates.
Figure 3: Real-world evaluation of ECL with MiniVLA and π0.5 on AGILEX PiPER under long-tailed task distributions. Evaluation amd results on xArm6 are provided in Appendix B .
AGILEX PIPER
xArm6
MiniVLA
π0.5
MiniVLA
π0.5
Task
APA
ECL(Ours)
APA
ECL(Ours)
APA
ECL(Ours)
APA
ECL(Ours)
Task 0
52 ±7.1
68 ±6.6
42 ±7.0
42 ±7.0
60 ±6.9
62 ±6.9
64 ±6.8
68 ±7.0
Task 1
18 ±5.4
26 ±6.2
36 ±6.8
38 ±6.9
20 ±5.7
38 ±6.9
20 ±5.7
40 ±6.9
Task 2
10 ±4.2
16 ±5.2
18 ±5.4
20 ±5.7
12 ±4.6
20 ±5.7
22 ±5.9
24 ±6.0
Task 3
4 ±2.8
6 ±3.4
16 ±5.2
18 ±5.4
4 ±2.8
6 ±2.8
14 ±4.9
20 ±5.7
Table 2: Real-world per-task success rates (%) on AGILEX PIPER and xArm6. For each platform, APA and ECL are evaluated with MiniVLA and π0.5 on six tasks ordered from head tasks (Task 0-1) to tail tasks (Task 2-5).
Figure 4: Ablation studies of ECL on (a) the training coefficient λ , (b) visual-token selection strategies, (c) the counterfactual masking ratio K , and (d) the masked-branch supervision weight μ . Performance is measured by the average success rate (%) across 10 tasks in LIBERO-Core-LT using MiniVLA.
Figure 5: Robustness of ECL to context-level long-tailed distributions in simulation (left) and the real world (right). Contexts 1-5 correspond to the scenes below each plot. Yellow and teal lines and bars show success rates (%) for the LTC baseline and ECL respectively. Pink shading marks rare or unseen contexts (2-5).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Language instruction
FULL
LT
0
Pick up the black bowl next to the plate and place it on the plate
46
46
1
Pick up the black bowl next to the cookie box and place it on the plate
47
28
2
Pick up the black bowl on the cookie box and place it on the plate
45
19
3
Pick up the ketchup and place it in the basket
42
15
4
Pick up the alphabet soup and place it in the basket
47
11
5
Push the plate to the front of the stove
39
9
Appendix
Table 3: LIBERO-Core task definitions and demonstration counts. “FULL” and “LT” denote the full and long-tailed training configurations, respectively. Task IDs follow the ordering in the project benchmark map.
Task
Language instruction
FULL
LT
0
Pick up the block and place it in the box
50
50
1
Stack the red block on top of the blue block
48
46
2
Stack the two blocks on the left on top of the block on the right
47
29
3
Insert the three-prong plug into the socket
49
13
4
Push the box to the designated location
46
10
5
Pick up the water bottle and place it in the box
45
9
Appendix
Table 4: Real-world task definitions and demonstration allocation. FULL denotes the approximately balanced reference configuration, and LT denotes the long-tailed configuration. Task IDs follow the order used in the real-world success-rate table.
Figure 6: Examples from the real-world task-level long-tailed dataset. AGILEX PIPER provides global-camera and wrist-camera observations, while xArm6 provides global-camera observations. The columns show the six manipulation tasks listed in Table 4 .
Figure 7: Examples from the Long-Tailed Context (LTC) dataset. Columns correspond to Scenes 0–5 in Table 5 ; the top and bottom rows show the global-camera and wrist-camera observations, respectively.
Language instruction
Scene
FULL
LT
Put the green block into the box
0
50
50
1
50
25
2
50
20
3
50
15
4
50
10
5
50
5
Appendix
Table 5: Context-level task definition and demonstration counts used in Section 4.4. All scenes share the same language instruction. FULL contains 50 demonstrations per scene, while LT progressively reduces the number of demonstrations for rarer scene configurations.
Backbone
Rollout limit
Episodes/task
Seeds
Episodes/seed
Action horizon
MiniVLA
350 steps
225
7, 14, 21
75
—
π0.5
350 steps
225
7, 14, 21
75
10 actions
Appendix
Table 6: Configuration for the simulation rollout evaluation.
Table 7: MiniVLA rollout for Task 3, “Pick up the ketchup and place it in the basket.”
Table 8: MiniVLA rollout for Task 6, “Push the plate to the front of the stove.”
Table 9: π0.5 rollout for Task 8, “Put the wine bottle on top of the cabinet.”
Table 10: π0.5 rollout for Task 9, “Put the wine bottle on the rack.”
Robot
Visual observations
Backbones
Comparison / trials
AGILEX PIPER
Global camera + wrist
MiniVLA, π0.5
APA vs. ECL / 50 per task
xArm6
Global camera
MiniVLA, π0.5
APA vs. ECL / 50 per task
Appendix
Table 11: Configuration for the real-world rollout evaluation.
Table 12: AGILEX PIPER rollout with MiniVLA (global-camera and wrist-camera views).
Table 13: AGILEX PIPER rollout with π0.5 (global-camera and wrist-camera views).
Figure 8: xArm6 per-task success rates for MiniVLA and π0.5 , comparing APA with ECL.
Table 14: xArm6 rollout comparison with MiniVLA.
Table 15: xArm6 rollout comparison with π0.5 .
Figure 9: Illustrative comparison of mask selection strategies. From left to right: object-only, action-only, random, and object–action relation selection. Peach overlays with white borders mark selected patches; all panels select 26/256 patches. The supplied visualization uses semantic-mask and saved-attention proxies, as described below, rather than a direct export of the complete ECL selector.
Figure 10: Visual evidence on representative LT tail tasks. Each square shows a paired comparison with the LT baseline on the left and ECL on the right. The top row shows “put the bowl on top of the cabinet,” and the bottom row shows “put the wine bottle on the rack.” Relative to the LT baseline, ECL more consistently grounds its decisions in instruction-relevant objects, interaction regions, and goal locations, supporting our finding that ECL effectively learns the correct task-specific visual evidence for tail-task decision making.
Method
Data size
GPUs
Batch
Logged steps
Mean (s)
Median (s)
Logged (h)
FULL
55,966
1
20/20
100,738
0.692
0.660
18.41
LT (Baseline)
19,234
1
20/20
34,622
0.608
0.570
5.89
APA
21,049
1
20/20
37,888
0.673
0.651
7.21
ECL (Ours)
19,234
1
20/20
34,622
1.068
1.090
10.27
Appendix
Table 16: Recorded MiniVLA training cost. GPU counts and batch sizes are taken from saved configurations. Batch denotes per-device/global batch size. Logged hours cover only the supplied timing records, which span different training intervals; these rows are not a controlled overhead comparison.
Vision-Language-Action (VLA) models are typically trained with behavior cloning (BC) on expert demonstrations. However, BC provides only positive supervision for expert actions, without explicit negative supervision indicating which actions are instruction-inconsistent or otherwise inappropriate. Reinforcement learning (RL) can provide such corrective signals, but often relies on externally specified rewards or curated non-expert data, both of which are costly to obtain in robotics. We show that offline RL for VLA models need not rely on curated non-expert trajectories: successful expert demonstrations alone can be transformed into dense corrective supervision through instruction relabeling. Specifically, by pairing expert actions with mismatched alternative instructions, we synthesize counterfactual instruction-observation-action tuples from the dataset and combine them with adversarial discriminator training to learn an instruction-grounded reward model for offline RL, without collecting additional rollouts or annotations. On the robustness-focused LIBERO-PRO benchmark, our method improves robustness to object position and task perturbations over a strong state-of-the-art baseline. It also outperforms competitive baselines in real-robot experiments on the TX-G2 (compatible with AGIBot G2). More broadly, our results suggest that, for data-constrained VLA learning, extracting denser supervision from each demonstration can complement collecting additional data.
Haru Kondoh, Kei Ota, Asako Kanezaki +1
Institute of Science Tokyo · AI Robot Association (AIRoA) · RIKEN AIP +1
Vision-language-action models (VLAs) are promising general-purpose robot policies, but adapting them to new tasks typically requires costly task-specific teleoperation data. As an alternative, we study one-shot demo-conditioned VLAs, where a robot policy is conditioned on a single demonstration video of an unseen task. We find that existing end-to-end approaches often struggle when successful execution requires precisely localizing small target regions. To address this limitation, we propose SeeTraceAct, a demo-conditioned VLA framework that encourages precise spatial grounding through visibility-aware prediction of future end-effector traces. To enable reproducible evaluation with cross-embodiment demonstrations, we introduce and release RoboCasa-DC, a demo-conditioned extension of RoboCasa that pairs its robot trajectories with newly collected humanoid demonstrations. Experiments on RoboCasa-DC and a real-world benchmark, where a Franka Panda arm is conditioned on human demonstrations, show that SeeTraceAct outperforms baselines, achieving the best success rate across all four RoboCasa-DC settings and improving real-world average success by 9.5 percentage points.
Jaehyeon Son, Junhyun Kim, Kyle Kam +7
Georgia Institute of Technology · Allen Institute for AI · Johns Hopkins University +1
Memory remains a critical bottleneck for long-horizon robotic manipulation, as standard Vision-Language-Action (VLA) policies often fail when task-relevant cues become occluded or unobservable over time. While existing memory-augmented methods utilize historical context, they either suffer from severe information bottlenecks, incur high latency via decoupled dual systems, or rely on unselective buffers that accumulate massive visual redundancies. To address these limitations, we introduce EventVLA, an end-to-end framework founded on the concept of sparse visual evidence memory that comprises two core components: foundational visual anchors to retain initial and short-term contexts, and a dynamic Keyframe Evidence Memory (KEM) module. Specifically, KEM directly predicts future keyframe probabilities from the VLA's latent embeddings to autonomously capture and store sparse, task-critical visual events. This foresight-driven mechanism empowers the policy to dynamically evaluate the future causal utility of current observations, preserving transient visual evidence before it becomes unobservable. Furthermore, we propose RoboTwin-MeM, a diagnostic benchmark specifically designed to evaluate non-Markovian manipulation tasks with interactive visual evidence. Extensive evaluations show that across 17 memory-requiring simulation tasks and 4 real-world bimanual tasks, EventVLA achieves an average success rate improvement of +40% over state-of-the-art memory-augmented VLAs.
Ganlin Yang, Zhangzheng Tu, Yuqiang Yang +10
University of Science and Technology of China · Shanghai AI Laboratory · Dalian University of Technology +5