When Does Cross-View Consistency Matter in Multi-View Appearance Augmentation? A Controlled Study for Visuomotor Policies
Organizations: College of Engineering, Peking University
Abstract
Multi-view visual augmentation often seeks consistency across cameras, but it is unclear which forms of consistency matter for downstream control. We use simulation-state replay to manipulate cross-view coupling while preserving the per-camera perturbation distributions, physical trajectories, and action labels. Across seven LIBERO manipulation tasks, marginal-matched changes in global appearance, ghost injection, and target color have small average costs for a late-fusion Diffusion Policy: point estimates are near zero or negative, and the largest upper endpoint of the 90% posterior credible intervals is 2.55 percentage points. In contrast, geometrically anti-coupled observations reduce success by 3.30 percentage points relative to coupled observations (90% CrI [0.64, 6.28]), with displacement magnitude, direction marginals, and the number of affected views matched. Camera interventions reveal substantial dependence on wrist-view geometry, while feature analysis shows appearance sensitivity patterns associated with the factors randomized during training. ACT shows a higher target-color cost estimate than DP, with uncertainty that includes zero and sensitivity to task composition. Within the tested tasks and policy configurations, these findings motivate selective consistency constraints: appearance coupling can sometimes be relaxed, whereas task-relevant geometry and cross-view spatial correspondence warrant protection.
Figures & tables
| Task | Instruction |
|---|---|
| 10_5 | Put both the alphabet soup and the tomato sauce in the basket. |
| 10_9 | Pick up the book and place it in the back compartment of the caddy. |
| goal_1 | Open the top drawer and put the bowl inside. |
| goal_6 | Put the cream cheese in the bowl. |
| goal_7 | Put the wine bottle on the rack. |
| spatial_2 | Pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate. |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Identifier | Definition |
|---|---|
| C0 | No augmentation; original observations only. |
| C1 / C3 / C5 | Global-appearance sharing with . |
| C6 / C7 | Ghost injection: both-or-neither versus exactly one camera. |
| C8 | C1 plus independent image-space photometric jitter. |
| C9 / C10 | Shared versus independent target-color switch and draw. |
| C11 | Offset-Wrist: visual target shifted 10 cm in the wrist view. |
| Task | d | d prime | Magnitude (cm) |
|---|---|---|---|
| 10_5 | 180.0 | 270.0 | 10 |
| 10_9 | 135.0 | 45.0 | 10 |
| goal_1 | 270.0 | 180.0 | 10 |
| goal_6 | 45.0 | 135.0 | 10 |
| goal_7 | 225.0 | 135.0 | 10 |
| spatial_2 | 45.0 | 135.0 | 10 |
| Contrast | Estimate [90% CrI] |
|---|---|
| Global appearance: linear trend, rho 1 to 0 | -0.64 [-3.43, 2.09] |
| Ghost: coupled minus single-view injection | 0.00 [-2.53, 2.55] |
| Target color: shared minus independent | -1.85 [-4.63, 0.69] |
| Additional photometric jitter: C1 minus C8 | -1.90 [-4.95, 0.95] |
| Estimand | Estimate [90% CrI] | Estimate [95% CrI] |
|---|---|---|
| c: Coupled - Anti | 3.30 [0.64, 6.28] | 3.30 [0.14, 6.94] |
| L: C1 - Coupled | 4.29 [1.61, 7.37] | 4.29 [1.19, 8.11] |
| Condition | 10_5 | 10_9 | goal_1 | goal_6 | goal_7 | spatial_2 | spatial_8 | Mean |
|---|---|---|---|---|---|---|---|---|
| C0 | 27.0 | 60.0 | 65.0 | 90.0 | 93.0 | 89.0 | 84.0 | 72.6 |
| C1 | 82.9 | 77.1 | 51.4 | 88.6 | 92.1 | 92.9 | 95.0 | 82.9 |
| C3 | 80.0 | 85.0 | 54.0 | 92.0 | 94.0 | 95.0 | 87.0 | 83.9 |
| C5 | 82.1 | 83.6 | 47.1 | 90.7 | 92.1 | 97.1 | 91.4 | 83.5 |
| C6 | 80.7 | 97.1 | 57.9 | 85.0 | 94.3 | 95.7 | 94.3 | 86.4 |
| C7 | 80.7 | 99.3 | 56.4 | 87.1 | 92.9 | 96.4 | 92.1 | 86.4 |
| Task | Base C1 | Coupled | Anti | Coupled - Anti (pp) |
|---|---|---|---|---|
| 10_5 | 82.9 | 45.0 | 40.7 | +4.3 |
| 10_9 | 77.1 | 94.3 | 90.7 | +3.6 |
| goal_1 | 51.4 | 65.0 | 60.0 | +5.0 |
| goal_6 | 88.6 | 60.7 | 57.1 | +3.6 |
| goal_7 | 92.1 | 90.0 | 81.4 | +8.6 |
| spatial_2 | 92.9 | 93.6 | 95.0 | -1.4 |
| Task | Batch difference |
|---|---|
| 10_5 | +5.0 |
| 10_9 | +32.5 |
| goal_1 | +5.0 |
| goal_6 | -7.5 |
| goal_7 | +5.0 |
| spatial_2 | +2.5 |
| Training | Policies | Mask third-person | Mask wrist |
|---|---|---|---|
| C0 | 35 | 26.7 [19.2, 33.6] | 54.9 [43.2, 63.4] |
| C1 | 49 | 15.3 [10.7, 19.9] | 56.2 [43.7, 65.7] |
| C5 | 49 | 16.5 [11.2, 21.7] | 56.1 [43.4, 65.7] |
| C9 | 49 | 21.1 [14.8, 27.0] | 59.5 [45.4, 70.4] |
| C10 | 49 | 19.4 [14.0, 24.7] | 57.5 [44.7, 66.9] |
| C11 | 35 | 32.8 [24.9, 39.7] | 42.9 [32.3, 51.5] |
| Task | 0 cm success (%) | 10 cm success (%) | Loss (pp) |
|---|---|---|---|
| 10_5 | 82.1 | 2.1 | 80.0 |
| 10_9 | 95.7 | 77.1 | 18.6 |
| goal_1 | 57.1 | 20.7 | 36.4 |
| goal_6 | 90.0 | 58.6 | 31.4 |
| goal_7 | 94.3 | 33.6 | 60.7 |
| spatial_2 | 96.4 | 80.0 | 16.4 |
| Contrast | Pairs | Estimate [90% CI] | Estimate [95% CI] |
|---|---|---|---|
| C12 - C11 | 35 | 0.7 [-2.2, 3.7] | 0.7 [-3.0, 4.4] |
| C1 - C12 | 35 | 4.3 [-8.2, 16.8] | 4.3 [-11.5, 20.0] |
| Element | Direction | Records | Source visible | Both direct | Projected |
|---|---|---|---|---|---|
| ghost | A->W | 356 | 69.9 | - | 12.1 |
| ghost | W->A | 328 | 16.5 | - | 13.4 |
| all_surfaces | W->A | 684 | 100.0 | 100.0 | 99.7 |
| floor | W->A | 684 | 16.8 | 9.1 | 0.3 |
| table | W->A | 684 | 100.0 | 100.0 | 99.6 |
| wall | W->A | 684 | 9.5 | 9.5 | 4.8 |
| Policy | Training | Global appearance | Target color |
|---|---|---|---|
| DP | C0 | 0.216 | 0.213 |
| DP | C1 | 0.036 | 0.157 |
| DP | C5 | 0.042 | 0.160 |
| DP | C9 | 0.037 | 0.048 |
| DP | C10 | 0.036 | 0.048 |
| ACT | C9 | 0.164 | 0.144 |
| Training | T0 (%) | Shared (%) | Indep. (%) | Cost [90% CrI] |
|---|---|---|---|---|
| C0 | 86.6 | 86.3 | 86.6 | -0.28 [-3.08, 2.46] |
| C1 | 87.2 | 86.3 | 87.3 | -1.05 [-3.64, 1.38] |
| C5 | 87.1 | 85.7 | 87.7 | -2.05 [-4.80, 0.37] |
| C8 | 88.7 | 87.4 | 88.4 | -0.99 [-3.79, 1.64] |
| Training | Estimate [95% CrI] |
|---|---|
| C0 | -0.28 [-3.67, 3.08] |
| C1 | -1.05 [-4.25, 1.82] |
| C5 | -2.05 [-5.40, 0.85] |
| C8 | -0.99 [-4.47, 2.18] |
| Task | Augmentation gain | Global appearance |
|---|---|---|
| 10_5 | 48.3 [38.7, 57.5] | 0.6 [-6.5, 7.8] |
| 10_9 | 17.3 [7.1, 27.5] | -6.2 [-14.0, 1.3] |
| goal_1 | -11.4 [-21.9, -0.8] | 4.4 [-5.1, 13.8] |
| goal_6 | -0.4 [-7.1, 6.8] | -2.2 [-8.0, 3.6] |
| goal_7 | 0.2 [-5.4, 6.1] | -0.1 [-5.2, 5.0] |
| spatial_2 | 4.8 [-1.3, 11.7] | -4.2 [-8.8, -0.2] |
| Task | Ghost | Added photo. jitter | Target color |
|---|---|---|---|
| 10_5 | -0.1 [-7.6, 7.4] | 4.9 [-3.0, 13.2] | 8.0 [-0.3, 16.3] |
| 10_9 | -2.0 [-4.9, 0.5] | -14.4 [-21.6, -7.2] | -7.7 [-14.0, -1.5] |
| goal_1 | 1.4 [-8.3, 11.0] | -4.0 [-14.5, 6.5] | -8.5 [-17.9, 1.2] |
| goal_6 | -2.2 [-9.1, 4.7] | 3.5 [-3.6, 10.9] | 1.5 [-5.0, 8.1] |
| goal_7 | 1.4 [-3.3, 6.3] | -3.7 [-8.5, 1.2] | -4.9 [-11.6, 1.6] |
| spatial_2 | -0.7 [-4.6, 3.1] | -2.1 [-7.0, 3.2] | -1.3 [-4.4, 1.5] |
| Configuration | C1 - Offset-Wrist | Shared - independent color |
|---|---|---|
| DP | 5.50 [2.28, 9.03] | -1.85 [-4.63, 0.69] |
| ACT | 8.50 [4.60, 12.93] | 2.19 [-0.52, 5.28] |
| Contrast | Pairs | Estimate [90% CI] | Estimate [95% CI] |
|---|---|---|---|
| C9 - C1 | 49 | -2.0 [-3.4, -0.7] | -2.0 [-3.7, -0.4] |
| C10 - C1 | 49 | 0.5 [-2.6, 3.7] | 0.5 [-3.4, 4.5] |
| C9 - C10 | 49 | -2.6 [-5.7, 0.6] | -2.6 [-6.6, 1.5] |
| ID | Estimate | 90% CI | 95% CI | p |
|---|---|---|---|---|
| H0a | 5.6 | [3.2, 8.2] | [2.8, 8.7] | 4.6e-05 |
| H0b | 9.3 | [6.9, 11.9] | [6.4, 12.4] | 6.6e-13 |
| H0c | 9.0 | [6.5, 11.6] | [6.1, 12.2] | 2.1e-11 |
| H1 | -0.7 | [-2.4, 1.1] | [-2.7, 1.4] | 1 |
| H2 | 0.0 | [-2.5, 2.5] | [-2.9, 2.9] | 1 |
| H3 | -2.0 | [-4.1, 0.3] | [-4.5, 0.7] | 1.1e-13 |
| ID | Population mean [90% interval] | New-task prediction [90% interval] |
|---|---|---|
| H0a | 6.4 [-2.9, 17.8] | 6.4 [-17.9, 41.1] |
| H1 | -0.5 [-4.1, 3.2] | -0.5 [-7.3, 7.1] |
| H2 | 0.1 [-2.9, 3.2] | 0.1 [-5.1, 5.8] |
| H3 | -1.4 [-6.6, 4.3] | -1.3 [-14.2, 12.1] |
| H5 | -1.7 [-5.9, 2.6] | -1.8 [-12.0, 7.5] |
| H6b | 2.3 [-1.1, 6.1] | 2.2 [-3.0, 10.0] |
| ID | Mean difference (pp) | Two-sided p |
|---|---|---|
| H0a | 5.0 | 0.438 |
| H0b | 8.4 | 0.0625 |
| H0c | 8.0 | 0.0156 |
| H1 | -0.6 | 0.75 |
| H2 | 0.0 | 1 |
| H3 | -2.4 | 0.5 |
| ID | Estimate range | Max. 90% upper | Omitted task | Covers zero |
|---|---|---|---|---|
| H0a | 1.58 to 9.29 | 13.83 | 10_9 | 1/7 |
| H0b | 6.79 to 11.48 | 16.59 | goal_1 | 0/7 |
| H0c | 5.59 to 10.29 | 15.78 | goal_1 | 0/7 |
| H1 | -1.48 to 0.37 | 3.30 | 10_9 | 7/7 |
| H2 | -0.36 to 0.37 | 3.25 | 10_9 | 7/7 |
| H3 | -3.26 to 0.32 | 3.42 | 10_9 | 6/7 |
| ID | Historical VB [90%] | glmer [90% CI] | brms [90% CrI] |
|---|---|---|---|
| H0a | 5.63 [3.24, 8.17] | 5.64 [2.27, 9.25] | 5.50 [2.28, 9.03] |
| H0b | 9.31 [6.88, 11.92] | 9.19 [5.17, 13.23] | 9.07 [5.29, 13.36] |
| H0c | 9.00 [6.53, 11.63] | 8.88 [4.42, 13.16] | 8.50 [4.60, 12.93] |
| H1 | -0.70 [-2.38, 1.08] | -0.64 [-3.17, 1.98] | -0.64 [-3.43, 2.09] |
| H2 | 0.01 [-2.46, 2.45] | 0.00 [-2.46, 2.35] | 0.00 [-2.53, 2.55] |
| H3 | -1.99 [-4.08, 0.25] | -1.90 [-4.97, 0.99] | -1.90 [-4.95, 0.95] |
| Item | Implementation and interpretation |
|---|---|
| Task/data scale | Ten planned tasks became seven; demonstrations per task changed from 20 to 15. Generalization and data-scale claims are limited accordingly. |
| Policy inventory | About 1,080 planned base policies became 784. The later geometry experiment adds 112, yielding 896. |
| Wrist displacement | The positive-control displacement increased from 5 to 10 cm following the pilot rule; conclusions concern the implemented magnitude. |
| Main estimator | A planned glmer analysis was initially implemented with VB; the present main effects use full MCMC. Earlier intervals remain separate. |
| ACT training budget | The budget changed from 50k to 25k steps after a half-budget check met its threshold. Results concern the actual 25k configuration. |
| Augmentation gate | The early common-seed estimate was 9.7 pp, below the planned 10 pp threshold. The original disposition record for that case is incomplete; the later 10.0 pp fixed-task estimate does not retroactively pass the gate. |
| Failure label | Units | Controlled counterpart |
|---|---|---|
| Task-object appearance | 10 | Recoloring |
| Ghost / missing object | 8 | Ghost injection |
| Global appearance | 6 | Appearance recipes |
| Photometric | 3 | White balance / brightness |
| Geometry | 2 | Visual target offset |
| Robot distortion | 6 | Not covered |
Multi-View Consistency
Multi-view consistency studies how visual models keep geometry, appearance, identity, and predictions aligned across different views of the same scene.
See all 202 papers