Beyond Viewpoint Generalization: What Multi-View Demonstrations Offer and How to Synthesize Them for Robot Manipulation?

Boyang Cai; Qiwei Liang; Jiawei Li; Shihang Weng; Zhaoxin Zhang; Tao Lin; Xiangyu Chen; Wenjie Zhang

cs.ROMar 23, 2026

Beyond Viewpoint Generalization: What Multi-View Demonstrations Offer and How to Synthesize Them for Robot Manipulation?

Authors: Boyang Cai, Qiwei Liang, Jiawei Li, Shihang Weng, Zhaoxin Zhang, Tao Lin, Xiangyu Chen, Wenjie Zhang, +6 more

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University · Beijing Jiaotong University · The Chinese University of Hong Kong, Shenzhen

Abstract

Does multi-view demonstration truly improve robot manipulation, or merely enhance cross-view robustness? We present a systematic study quantifying the performance gains, scaling behavior, and underlying mechanisms of multi-view data for robot manipulation. Controlled experiments show that, under both fixed and randomized backgrounds, multi-view demonstrations consistently improve single-view policy success and generalization. Performance varies non-monotonically with view coverage, revealing effective regimes rather than a simple "more is better" trend. Notably, multi-view data breaks the scaling limitation of single-view datasets and continues to raise performance ceilings after saturation. Mechanistic analysis shows that multi-view learning promotes manipulation-relevant visual representations, better aligns the action head with the learned feature distribution, and reduces overfitting. Motivated by the importance of multi-view data and its scarcity in large-scale robotic datasets, as well as the difficulty of collecting additional viewpoints in real world settings, we propose RoboNVS, a geometry-aware self-supervised framework that synthesizes novel-view videos from monocular inputs. The generated data consistently improves downstream policies in both simulation and real-world environments.

Figures & tables

Figure 1: Experimental setup overview. Camera layout, example multi-view observations, and the Clean/Random environment modes.

Figure 2: Comparison between single-view and multi-view training under clean and random settings. Multi-view training improves performance across tasks, with stronger gains under the random setting.

Setting	Seen Tasks									Unseen Tasks			Average
Setting	AB	BBH	CA	MCP	MPA	OL	PCP	PS	SBH	SHB	CB	PEC	Seen	Unseen	Overall
Single-view	52	16	46	16	10	24	40	24	64	68	48	0	32.4	38.7	34.0
Multi-view	68	28	40	30	14	28	52	34	72	80	50	8	40.7 (+8.3)	46.0 (+7.3)	42.0 (+8.0)

Table 1: Multi-view supervision improves performance of a multi-task VLA model on both seen and unseen tasks.

Task	$0^{\circ}$	$\pm 10^{\circ}$	$\pm 20^{\circ}$	$\pm 30^{\circ}$	$\pm 40^{\circ}$	$\pm 50^{\circ}$	$\pm 60^{\circ}$
AB	8	42	80	70	64	56	48
BBH	14	22	32	22	20	12	8
CA	24	32	62	48	50	18	12
CB	4	20	58	30	28	18	10
Mean	12.5	29.0	58.0	42.5	40.5	26.0	19.5

Table 2: Task-wise success rates (%) under each viewpoint offset in the standard offset study.

Task	$0^{\circ}$	$\pm 10^{\circ}$	$\pm 20^{\circ}$	$\pm 30^{\circ}$	$\pm 40^{\circ}$	$\pm 50^{\circ}$	$\pm 60^{\circ}$	$0^{\circ}+\pm 10^{\circ}$ Stack
AB	8	50	58	48	44	44	42	42
BBH	14	18	22	14	12	12	8	22
CA	24	22	32	26	22	18	14	32
CB	4	14	22	16	14	10	8	20
Mean	12.5	26.0	33.5	26.0	23.0	21.0	18.0	29.0

Table 3: Strict budget-matched VA experiment under fixed

0^{\circ}

evaluation. All budget-matched settings use the same data budget and training steps; the rightmost column reports the full three-view stack for reference.

Figure 3: Scaling laws of single-view vs. multi-view training on manipulation tasks. Multi-view data continues to improve performance when single-view training saturates, achieving comparable or higher performance with substantially less scene diversity.

Figure 4: Scaling behavior of multi-view demonstrations under fixed

0^{\circ}

evaluation. Each subplot shows success rate versus demonstrations per added view (

10\!\rightarrow\!50

) for one task. The black dashed line indicates the single-view baseline trained on the origin view.

Figure 5: Grad-CAM and input-gradient analysis under fixed $0^{\circ}$ evaluation. Multi-view supervision ( $0^{\circ}+\pm 20^{\circ}$ ) shifts attention toward the end-effector and manipulated objects, while single-view models exhibit stronger background sensitivity.

	Linear Probe ( $R^{2}\downarrow$ )						Nonlinear Probe ( $R^{2}\uparrow$ )
Setting	Adjust Bottle		Beat Block Hammer		Click Bell		Adjust Bottle		Beat Block Hammer		Click Bell
	End-effector	Object	End-effector	Object	End-effector	Object	End-effector	Object	End-effector	Object	End-effector	Object
Single-view	0.510	0.502	0.816	0.678	0.930	0.952	-0.112	-0.032	-1.220	-0.777	0.283	0.529
Multi-view	-0.116	-0.116	0.668	0.540	0.595	0.736	0.601	0.166	0.428	0.511	0.707	0.835
$\Delta$	-0.626	-0.618	-0.148	-0.138	-0.335	-0.216	+0.713	+0.198	+1.648	+1.288	+0.424	+0.306

Table 4: Clean-only probing results (

R^{2}

) for manipulation-relevant states under fixed

0^{\circ}

evaluation. Linear probing tests direct linear decodability (

\downarrow

), while nonlinear probing tests recoverability with a nonlinear decoder (

\uparrow

\Delta

denotes Multi-view

-

Single-view.

Metric	Random $\downarrow$				Clean $\downarrow$
	BBH		CA		BBH		CA
	Single	Multi	Single	Multi	Single	Multi	Single	Multi
MSE	0.0128	0.0070	0.0133	0.0087	0.0104	0.0067	0.0082	0.0074

Table 5: Policy-head MSE loss under frozen DINOv2 features (

\downarrow

lower is better).

Figure 6: Training dynamics of a multi-task VLA model under single-view (left) and multi-view (right) supervision. Success rates are reported over training steps (2k–5k) across six tasks. Best checkpoints are highlighted.

Method	AB	BBH	CA	CB	Method	AB	BBH	CA	CB
$0^{\circ}$	8	14	24	4	Crop	4	8	16	2
Perspective	6	2	6	2	Blur	8	4	8	4
Rotation	4	4	8	2	Mixed	12	16	6	4
View Aug. ( $\pm 20^{\circ}$ )	34	20	30	18

Table 6: Budget-matched policy success rates of conventional 2D image augmentations versus multi-view augmentation, evaluated at the same fixed

0^{\circ}

test viewpoint (SR%).

Figure 7: Pipeline of RoboNVS. A monocular manipulation video is first converted into a temporally consistent and metric-aligned 3D representation. The resulting DW-Mesh is rendered under target camera trajectories to produce RGB projections and occlusion masks. These geometric priors condition a diffusion-based video completion model to synthesize physically plausible novel-view demonstrations.

Method	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	VBench $\uparrow$
ZeroNVS-ft	0.3305	0.527	64.59	0.578
ReCamMaster	0.3865	0.451	68.16	0.585
CogNVS	0.4029	0.502	76.87	0.593
TrajectoryCrafter	0.3849	0.411	48.42	0.579
EX-4D	0.4013	0.406	42.66	0.588
RoboNVS w/o Bi-Mask	0.4272	0.369	35.90	0.593

Table 7: Visual quality comparison of RoboNVS and baseline models.

\uparrow

indicates higher is better and

\downarrow

indicates lower is better.

Figure 8: Qualitative comparison with generative baselines. RoboNVS produces more geometrically consistent and temporally stable novel-view videos. Baseline methods often exhibit object duplication, distorted robot structures, or incomplete mask filling.

Figure 9: Real-world setup and manipulation tasks. We evaluate RoboNVS on Bell Pushing, Fruit Pick-and-Place, and Lego Pick-and-Place using a Franka Panda robot.

Figure 10: Qualitative comparison on real-world tasks. Depth alignment improves target-view geometric masks, while RoboNVS further improves video completion quality, reconstructing robot arms and end-effectors more completely than EX-4D variants.

Method	Bell Pushing	Fruit Pick-and-Place	Lego Pick-and-Place
Monocular only	5/20 (25%)	2/20 (10%)	1/20 (5%)
EX-4D	8/20 (40%)	6/20 (30%)	4/20 (20%)
EX-4D w/ DA	10/20 (50%)	8/20 (40%)	7/20 (35%)
RoboNVS	14/20 (70%)	12/20 (60%)	11/20 (55%)

Table 8: Policy success rates on real-world tasks. Each result is evaluated over 20 trials.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Source Family	Datasets
OXE: large-scale robot datasets	Droid [ 28 ] , RT-1 [ 6 ] , BridgeData V2 [ 48 ]
OXE: specialized manipulation datasets	mimicplay [ 51 ] , ucsd_kitchen [ 57 ] , ur5_columbia_cairlab_pusht_real [ 12 ]
	utokyo_xarm_pick_and_place [ 37 ] , nyu_franka_play_dataset [ 13 ] , HYDRA [ 2 ]
Additional robot datasets	RH20T [ 15 ] , RoboTwin [ 11 ] , RoboCoin [ 55 ]

Appendix

Table 9: Training data sources used for RoboNVS. The corpus is primarily built from OXE subsets, with additional robotic manipulation datasets added for broader embodiment coverage.

Figure 11: Dataset composition used for training RoboNVS. The training corpus contains approximately 10,000 videos, with Droid forming the largest subset due to its scene diversity and broad manipulation coverage.

Category	Robot Embodiments
Standard arms	Franka, UR5, Sawyer, Google Robot, WidowX, Xarm
Mobile / dual-arm / multi-purpose platforms	Unitree G1 edu-u3, Mobile Aloha, A1X, AIRBOT_MMK2, alpha_bot_2
	AgilexCobotMagic, AgilexSplitALOHA, Galaxea R1 Lite

Appendix

Table 10: Robotic embodiments and platforms covered by the aggregated training corpus.

Diffusion Policy Training Setup
Observation modality	head_camera + agent_pos	Image encoder	ResNet-18
Image resolution	$256\times 192$	Batch size	128
Observation steps ( $n_{\text{obs}}$ )	3	Optimizer	AdamW
Prediction horizon	8	Learning rate	$1\times 10^{-4}$
Executed actions ( $n_{\text{action}}$ )	6	LR schedule	cosine decay
Diffusion type	DDPM	Weight decay	$1\times 10^{-6}$

Appendix

Table 11: Key training hyperparameters for Diffusion Policy (DP).

$\pi_{0}$ Training Setup
Backbone VLM	PaliGemma 2B + LoRA	Precision	bfloat16
Vision encoder	SigLIP	Parallelization	FSDP (4 GPUs)
Action expert	Gemma 300M + LoRA	Optimizer	AdamW
Peak learning rate	$2.5\times 10^{-5}$	Final learning rate	$2.5\times 10^{-6}$
LR schedule	cosine decay	Warmup steps	1000
Batch size	32	Gradient clipping	1.0

Appendix

Table 12: Key training hyperparameters for

\pi_{0}

Figure 12: Visualization of stochastic initializations. Each row shows one representative task, and each column shows a different randomized evaluation rollout. The figure highlights variations in object poses and tabletop textures.

Figure 13: Visualization of the 12 manipulation tasks in RoboTwin.

View Set	AB	BBH	CA	CB	Mean
$0^{\circ}$	8	14	24	4	12.5
$0^{\circ}+\{\pm 10^{\circ}\}$	54	30	46	18	37.0
$0^{\circ}+\{\pm 10^{\circ},\pm 20^{\circ}\}$	60	48	58	30	49.0
$0^{\circ}+\{\pm 10^{\circ},\pm 20^{\circ},\pm 30^{\circ}\}$	54	40	50	28	43.0
$0^{\circ}+\{\pm 10^{\circ},\ldots,\pm 40^{\circ}\}$	52	28	44	24	37.0
$0^{\circ}+\{\pm 10^{\circ},\ldots,\pm 50^{\circ}\}$	36	26	30	18	27.5

Appendix

Table 13: Success rates (%) under progressively stacked multi-view training. Starting from

0^{\circ}

, symmetric viewpoint pairs are incrementally added to the training set, while all policies are evaluated at the fixed

0^{\circ}

viewpoint.

View Set	AB	BBH	CA	CB	Mean
$0^{\circ}$	52	30	20	14	29.0
$0^{\circ}+\{\pm 10^{\circ},\pm 20^{\circ},\pm 30^{\circ}\}$	68	38	38	20	41.0
$0^{\circ}+\{\pm 10^{\circ},\ldots,\pm 60^{\circ}\}$	58	34	26	16	33.5

Appendix

Table 14: VLA performance under progressively stacked multi-view training. The

\pi_{0}

policy is trained with the same experimental configuration as the main-text experiments, while all policies are evaluated at the fixed

0^{\circ}

viewpoint. (SR%)

View Set	AB	BBH	CA	CB	Mean
$0^{\circ}$	8	14	24	4	12.5
$0^{\circ}+\{\pm 10^{\circ}\}$	24	18	26	16	21.0
$0^{\circ}+\{\pm 10^{\circ},\pm 20^{\circ}\}$	34	20	30	18	25.5
$0^{\circ}+\{\pm 10^{\circ},\pm 20^{\circ},\pm 30^{\circ}\}$	28	16	28	12	21.0
$0^{\circ}+\{\pm 10^{\circ},\ldots,\pm 40^{\circ}\}$	26	16	26	8	19.0
$0^{\circ}+\{\pm 10^{\circ},\ldots,\pm 50^{\circ}\}$	18	14	24	8	16.0

Appendix

Table 15: Budget-matched progressive multi-view stacking under fixed

0^{\circ}

evaluation. (SR%)

ACT Policy Configuration
Backbone	ResNet-18	Hidden dim	512
Encoder layers	4	Decoder layers	7
Attention heads	8	Feedforward dim	3200
Dropout	0.1	Position embedding	sine
Batch size	16	Training epochs	6000
Learning rate	$1\times 10^{-5}$	Backbone LR	$1\times 10^{-5}$

Appendix

Table 16: Key architectural and training hyperparameters for ACT.

Task	$0^{\circ}$	$\pm 10^{\circ}$	$\pm 20^{\circ}$	$\pm 30^{\circ}$
AB	28	48	50	34
BBH	14	16	24	16
CA	12	16	16	14
CB	0	2	6	0
SBH	28	34	50	46
MPA	8	10	12	8

Appendix

Table 17: Success rate comparison of ACT policies under multi-view vs. single-view training. ACT policies trained with multi-view demonstrations consistently outperform the single-view baseline under fixed

0^{\circ}

evaluation. The best performance generally appears under moderate viewpoint offsets around

\pm 20^{\circ}

Figure 14: View augmentation under alternative base viewpoints. (a) The two base viewpoints used in this study. (b) Task success rates when the base view is $40^{\circ}$ or $90^{\circ}$ . In both settings, multi-view training improves performance over single-view training.

Figure 15: Cross-view robustness evaluation. Policies trained with multi-view demonstrations substantially outperform single-view policies across four off-center test viewpoints. This confirms that multi-view supervision also improves robustness when the test camera moves.

Figure 16: View augmentation with DP3 using point-cloud observations. (a) Example point-cloud observations from the frontal $0^{\circ}$ view. (b) DP3 success rates under progressively added viewpoints. Multi-view training improves performance even when the policy uses 3D point clouds instead of RGB images.

Policy Architecture		Training Setup
Policy type	DP3 (DDIM Diffusion)	Optimizer	AdamW
Point cloud encoder	PointTransformer	Learning rate	$1\times 10^{-4}$
Transformer layers	2	Batch size	16
Attention heads	2	Training epochs	3000
Encoder output dim	128	Weight decay	$1\times 10^{-6}$
UNet channels	[512, 1024, 2048]	LR schedule	cosine

Appendix

Table 18: Key architectural and training hyperparameters for DP3.

Figure 17: Qualitative comparison of generative models under a $+20^{\circ}$ rotation in RoboTwin. All mask-based methods are conditioned on identical masks derived from simulator ground-truth depth. RoboNVS reconstructs robot joints and object geometry more faithfully than the compared generators.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
CogNVS	17.154	0.7169	0.3621	245.13	47.36
EX-4D	17.031	0.6703	0.4331	248.26	79.26
TrajectoryCrafter	16.480	0.6687	0.4714	217.33	68.47
ZeroNVS-ft	14.350	0.7065	0.5465	215.58	86.18
RoboNVS	18.839	0.7575	0.3018	132.87	33.28

Appendix

Table 19: Quantitative comparison of generative video synthesis quality in RoboTwin.

Training Data	Click Alarm Clock	Click Bell
Single-view baseline ( $0^{\circ}$ only)	19/50 (38%)	5/50 (10%)
EX-4D + $0^{\circ}$	36/50 (72%)	9/50 (18%)
TrajectoryCrafter + $0^{\circ}$	28/50 (56%)	9/50 (18%)
CogNVS + $0^{\circ}$	34/50 (68%)	35/50 (70%)
ZeroNVS-ft + $0^{\circ}$	22/50 (44%)	6/50 (12%)
RoboNVS + $0^{\circ}$	40/50 (80%)	38/50 (76%)

Appendix

Table 20: Policy success rates after multi-view data augmentation using different generative models. All policies are evaluated at the RoboTwin

0^{\circ}

viewpoint over 50 trials.

Figure 18: Qualitative comparison of mask-based generative inpainting. All models receive the same mask sequences. RoboNVS produces more complete robot structures, fewer ghosting artifacts, and fewer geometric distortions around the arm and the contacted object.

Figure 19: Qualitative comparison of DW-Mesh renderings constructed from different video depth prediction methods. The first row shows source views from three manipulation scenes. Each column visualizes DW-Mesh rendering from a different depth strategy. Brown boxes compare temporal depth estimators, red boxes compare metric 3D-aware models, and yellow boxes highlight why DepthCrafter w/ DA3 is cleaner than the other aligned variant.

Explore similar work

Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
Apr 1, 2026Yichen Xie, Yixiao Wang, Shuqi Zhao +4Robot Imitation Learning Imitation Learning
UC Berkeley · Lambda, Inc · MIT
VistaBot: View-Robust Robot Manipulation via Spatiotemporal-Aware View Synthesis
Apr 23, 2026Songen Gu, Yuhang Zheng, Weize Li +6Generalization in Robotic Manipulation Novel View Synthesis
1Fudan University · 2TARS Robotics · 4NUS +1
Self Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation
Aug 1, 2026Andres Rivas, Anselmo R. Cukla, Rodrigo S. Guerra +2Robotic Grasping Robot Skill Learning
Robotics and AI Lab, Technological University of Uruguay, Rivera, Uruguay · Center of Technology, Federal University of Santa Maria, Santa Maria, Brazil · Computer Science Center, Federal University of Rio Grande, Rio Grande, Brazil

Apr 1, 2026J/K move · Enter open · S save

Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning

Yichen Xie, Yixiao Wang, Shuqi Zhao, Cheng-En Wu+3

UC Berkeley · Lambda, Inc · MIT

Robot Imitation Learning Imitation Learning Data-Efficient Robot Learning Generalization in Robotic Manipulation

Open paper AskPDF arXiv

The generalization ability of imitation learning policies for robotic manipulation is fundamentally constrained by the diversity of expert demonstrations, while collecting demonstrations across varied environments is costly and difficult in practice. In this paper, we propose a practical framework that exploits inherent scene diversity without additional human effort by scaling camera views during demonstration collection. Instead of acquiring more trajectories, multiple synchronized camera perspectives are used to generate pseudo-demonstrations from each expert trajectory, which enriches the training distribution and improves viewpoint invariance in visual representations. We analyze how different action spaces interact with view scaling and show that camera-space representations further enhance diversity. In addition, we introduce a multiview action aggregation method that allows single-view policies to benefit from multiple cameras during deployment. Extensive experiments in simulation and real-world manipulation tasks demonstrate significant gains in data efficiency and generalization compared to single-view baselines. Our results suggest that scaling camera views provides a practical and scalable solution for imitation learning, which requires minimal additional hardware setup and integrates seamlessly with existing imitation learning algorithms. The website of our project is https://yichen928.github.io/robot_multiview.

Abstract

Explore similar work

Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning

VistaBot: View-Robust Robot Manipulation via Spatiotemporal-Aware View Synthesis

Self Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation