Multi-view visual augmentation often seeks consistency across cameras, but it is unclear which forms of consistency matter for downstream control. We use simulation-state replay to manipulate cross-view coupling while preserving the per-camera perturbation distributions, physical trajectories, and action labels. Across seven LIBERO manipulation tasks, marginal-matched changes in global appearance, ghost injection, and target color have small average costs for a late-fusion Diffusion Policy: point estimates are near zero or negative, and the largest upper endpoint of the 90% posterior credible intervals is 2.55 percentage points. In contrast, geometrically anti-coupled observations reduce success by 3.30 percentage points relative to coupled observations (90% CrI [0.64, 6.28]), with displacement magnitude, direction marginals, and the number of affected views matched. Camera interventions reveal substantial dependence on wrist-view geometry, while feature analysis shows appearance sensitivity patterns associated with the factors randomized during training. ACT shows a higher target-color cost estimate than DP, with uncertainty that includes zero and sensitivity to task composition. Within the tested tasks and policy configurations, these findings motivate selective consistency constraints: appearance coupling can sometimes be relaxed, whereas task-relevant geometry and cross-view spatial correspondence warrant protection.
Figures & tables
Figure 1: Controlled visual augmentation. (a) State replay preserves physical trajectories and action labels while rendering each camera under its assigned recipe. (b) Shared and independent joint distributions have the same per-camera marginals. (c) Same-state examples of shared appearance (C1) and independent recoloring (C10), from spatial_8 , demo 0, frame 0, seed 1, variant 1. The historical renders are rotated into policy-input orientation. The matched recoloring experiment compares shared recoloring (C9) with C10.
Task
Instruction
10_5
Put both the alphabet soup and the tomato sauce in the basket.
10_9
Pick up the book and place it in the back compartment of the caddy.
goal_1
Open the top drawer and put the bowl inside.
goal_6
Put the cream cheese in the bowl.
goal_7
Put the wine bottle on the rack.
spatial_2
Pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate.
Table 1: Seven LIBERO tasks. Short identifiers omit the libero_ prefix.
Figure 2: Average effects of appearance coupling and added photometric variation. Points and bars show posterior means and 90% CrI. (a) Coupling contrasts with matched per-camera perturbation distributions. (b) The effect of adding independent photometric jitter to C1. Positive values indicate lower success in the target condition; negative values favor it.
Figure 3: Marginal-matched geometry experiment. (a) Joint direction probabilities; every row and column sums to 0.5. (b) Posterior means, thick 90% CrI, and thin 95% CrI. The primary within-batch contrast c differs from the supporting cross-batch contrast L . (c) Descriptive success by task, with segments linking the two conditions. The two new conditions use 49 policies each.
Figure 4: Sensitivity to camera masking and wrist-view displacement. (a) Within-condition effects of masking one camera on the original scene. (b) E1 measures first exposure to a wrist target offset using paired 0/10 cm episodes and OSMesa rendering. Points and bars show posterior means and 90% CrI.
Figure 5: Co-visibility and factor-specific feature sensitivity. (a) Projected co-visibility by element and direction; A denotes third-person and W wrist. The orange mark gives pooled direct target visibility. Photometric coverage is 100% by definition. (b) DP feature-invariance ratios for two perturbation types, averaged over both cameras and seven seed-1 policies per condition. Lower values indicate smaller appearance-induced changes relative to state-induced changes.
Figure 6: Task heterogeneity and policy configuration. (a–d) Task-specific estimates from fixed task-interaction models. Panels (a–c) measure coupling effects; (d) measures added photometric variation. (e) Average target-color effects for DP and ACT. (f) Shared-appearance augmentation gain relative to C0. Bars denote 90% CrI. Panel (f) uses a wider horizontal range to accommodate the large gain on 10_5 .
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Identifier
Definition
C0
No augmentation; original observations only.
C1 / C3 / C5
Global-appearance sharing with ρ=1/0.5/0 .
C6 / C7
Ghost injection: both-or-neither versus exactly one camera.
C8
C1 plus independent image-space photometric jitter.
C9 / C10
Shared versus independent target-color switch and draw.
C11
Offset-Wrist: visual target shifted 10 cm in the wrist view.
Appendix
Table A.1: Condition inventory.
Task
d
d prime
Magnitude (cm)
10_5
180.0
270.0
10
10_9
135.0
45.0
10
goal_1
270.0
180.0
10
goal_6
45.0
135.0
10
goal_7
225.0
135.0
10
spatial_2
45.0
135.0
10
Appendix
Table A.2: Fixed task-wise geometry directions in world coordinates (degrees).
Contrast
Estimate [90% CrI]
Global appearance: linear trend, rho 1 to 0
-0.64 [-3.43, 2.09]
Ghost: coupled minus single-view injection
0.00 [-2.53, 2.55]
Target color: shared minus independent
-1.85 [-4.63, 0.69]
Additional photometric jitter: C1 minus C8
-1.90 [-4.95, 0.95]
Appendix
Table B.1: DP appearance contrasts. Positive values favor the reference; effects are in percentage points (pp).
Estimand
Estimate [90% CrI]
Estimate [95% CrI]
c: Coupled - Anti
3.30 [0.64, 6.28]
3.30 [0.14, 6.94]
L: C1 - Coupled
4.29 [1.61, 7.37]
4.29 [1.19, 8.11]
Appendix
Table B.2: Geometry effects with marginal-matched direction distributions (pp).
Condition
10_5
10_9
goal_1
goal_6
goal_7
spatial_2
spatial_8
Mean
C0
27.0
60.0
65.0
90.0
93.0
89.0
84.0
72.6
C1
82.9
77.1
51.4
88.6
92.1
92.9
95.0
82.9
C3
80.0
85.0
54.0
92.0
94.0
95.0
87.0
83.9
C5
82.1
83.6
47.1
90.7
92.1
97.1
91.4
83.5
C6
80.7
97.1
57.9
85.0
94.3
95.7
94.3
86.4
C7
80.7
99.3
56.4
87.1
92.9
96.4
92.1
86.4
Appendix
Table B.3: Descriptive DP success on held-out appearance B (%).
Figure S1: Global-appearance dose response. (a) Equal-weight seven-task descriptive average. (b) Individual-task success rates, with cell values in percent and a common 0–100% color scale. Columns follow C1, C3, and C5 as sharing probability ρ decreases from 1 to 0. All rates use their condition’s included seeds; no confidence or credible intervals are inferred.
Figure S2: Matched photometric perturbations at test time. (a) Posterior mean coupling costs with 90% CrI. (b) Descriptive success for unperturbed (T0), shared (T-shared), and independent (T-indep) modes, vertically offset within each training-condition row for readability. The success-rate axis spans 80–92%. Each policy has 20 episodes per mode on the original scene. Training C8 includes stronger independent photometric jitter; C0, C1, and C5 do not.
Task
Augmentation gain
Global appearance
10_5
48.3 [38.7, 57.5]
0.6 [-6.5, 7.8]
10_9
17.3 [7.1, 27.5]
-6.2 [-14.0, 1.3]
goal_1
-11.4 [-21.9, -0.8]
4.4 [-5.1, 13.8]
goal_6
-0.4 [-7.1, 6.8]
-2.2 [-8.0, 3.6]
goal_7
0.2 [-5.4, 6.1]
-0.1 [-5.2, 5.0]
spatial_2
4.8 [-1.3, 11.7]
-4.2 [-8.8, -0.2]
Appendix
Table B.13: Augmentation gain and global-appearance coupling: fixed task-interaction estimates (pp; 90% CrI).
Table C.5: Fitter sensitivity. Each interval retains the corresponding method’s interpretation.
Figure S3: Leave-one-task-out sensitivity. Each task row omits the named task. Points and bars are posterior means and 90% CrI on a common horizontal scale. The shaded Full set row shows the full-task posterior mean (diamond) and 90% CrI; dashed lines mark zero. These are sample-composition checks, not independent replications. The photometric contrast remains an added-perturbation control.
Item
Implementation and interpretation
Task/data scale
Ten planned tasks became seven; demonstrations per task changed from 20 to 15. Generalization and data-scale claims are limited accordingly.
Policy inventory
About 1,080 planned base policies became 784. The later geometry experiment adds 112, yielding 896.
Wrist displacement
The positive-control displacement increased from 5 to 10 cm following the pilot rule; conclusions concern the implemented magnitude.
Main estimator
A planned glmer analysis was initially implemented with VB; the present main effects use full MCMC. Earlier intervals remain separate.
ACT training budget
The budget changed from 50k to 25k steps after a half-budget check met its threshold. Results concern the actual 25k configuration.
Augmentation gate
The early common-seed estimate was 9.7 pp, below the planned 10 pp threshold. The original disposition record for that case is incomplete; the later 10.0 pp fixed-task estimate does not retroactively pass the gate.
Appendix
Table D.1: Material changes and departures from earlier plans.
Failure label
Units
Controlled counterpart
Task-object appearance
10
Recoloring
Ghost / missing object
8
Ghost injection
Global appearance
6
Appearance recipes
Photometric
3
White balance / brightness
Geometry
2
Visual target offset
Robot distortion
6
Not covered
Appendix
Table E.1: Generator pilot: counts out of ten paired-view units.
Figure S4: Cosmos-Transfer2.5 pilot observations. Top: counts of annotated categories among ten paired-view units, comprising five tasks under two prompt/seed schemes. Bottom: third-person and wrist frames at frame 46 of demonstration 0 for two tasks. The labels identify independent or shared prompts and seeds. Images are taken from the saved generator outputs. The gray bars correspond to robot distortion and temporal changes, which lie outside the static injection experiments.
Multi-view consistency studies how visual models keep geometry, appearance, identity, and predictions aligned across different views of the same scene.
Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns a policy's own successful episodes into targeted demonstrations for its failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory, ensuring geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance. These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization.
Visuomotor policies for manipulation have demonstrated remarkable potential in modeling complex robotic behaviors, yet minor alterations in the robot's initial configuration and unseen obstacles easily lead to out-of-distribution observations. Without extensive data collection effort, these result in catastrophic execution failures. In this work, we introduce an effective data augmentation framework that generates visually realistic fisheye image sequences and corresponding physically feasible action trajectories from real-world eye-in-hand demonstrations, captured with a portable parallel gripper with a single fisheye camera. We introduce a novel Gaussian Splatting formulation, adapted to wide FoV fisheye cameras, to reconstruct and edit the 3D scene with unseen objects. We utilize trajectory optimization to generate smooth, collision-free, view-rendering-friendly action trajectories and render visual observations from corresponding novel views. Comprehensive experiments in simulation and the real world show that our augmentation framework improves the success rate for various manipulation tasks in both the same scene and the augmented scene with obstacles requiring collision avoidance.
Chuer Pan, Litian Liang, Dominik Bauer +4
Stanford University · Columbia University · Toyota Research Institute
Vision-Language-Action (VLA) policies often rely strongly on the camera viewpoints seen during training, causing substantial performance degradation when deployed from unseen perspectives. Collecting demonstrations from sufficiently diverse physical viewpoints is expensive and still provides only sparse coverage of the viewpoint space. We introduce InfiNoVA, a data-augmentation framework that converts synchronized multi-camera demonstrations into a dense distribution of geometrically consistent training views. InfiNoVA reconstructs each manipulation trajectory as a time-varying 3D Gaussian representation and renders novel observations from sampled camera poses while preserving the original state-action correspondence. This explicit scene representation improves frame-level fidelity and temporal consistency while reducing task-critical hallucinations observed in generative novel-view synthesis. Across four real-world manipulation tasks, policies trained with InfiNoVA achieve 5.4x higher average success under unseen randomized viewpoints than both VISTA-based augmentation and the unaugmented policy. InfiNoVA further achieves 1.7x higher success than training directly on all five physical camera views. These results show that dense, geometrically grounded viewpoint augmentation provides a practical route toward camera-robust robot policies without modifying the underlying policy architecture.
Sai Puneeth Reddy Gottam, Elmar Rueckert, Vedant Dave
Chair of Cyber Physical Systems, Montanuniversität Leoben, Austria