Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

Wonbong Jang; Shikun Liu; Soubhik Sanyal; Juan Camilo Perez; Kam Woh Ng; Sanskar Agrawal; Juan-Manuel Perez-Rua; Yiannis Douratsos

cs.CVApr 10, 2026

Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

Authors: Wonbong Jang, Shikun Liu, Soubhik Sanyal, Juan Camilo Perez, Kam Woh Ng, Sanskar Agrawal, Juan-Manuel Perez-Rua, Yiannis Douratsos,

Organizations: Meta AI, London, United Kingdom

Abstract

Recovering camera parameters from images and rendering scenes from novel viewpoints have been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task depends on what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. To our knowledge, this is the first model to predict camera poses and do camera-controlled video generation within a single framework. We represent each camera as dense ray pixels (raxels), a pixel-aligned encoding that lives in the same latent space as video frames, and denoise the two jointly through a Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, generating video from input images along a pre-defined trajectory, and jointly synthesizing video and trajectory from input images. We evaluate on pose estimation and camera-controlled video generation, and introduce a closed-loop self-consistency test showing that the model's predicted poses and its renderings conditioned on those poses agree. Ablations against Plücker embeddings confirm that representing cameras in a shared latent space with video is subtantially more effective.

Figures & tables

Figure 2: Overview of the Rays as Pixels Framework. Training (left): We jointly encode video frames and their corresponding raxel images into a shared latent space using a frozen spatio-temporal VAE encoder. Video inputs undergo 4 $\times$ temporal compression, while the temporal dimensions of image inputs remain as they are. The DiT jointly denoises these latents via Decoupled Self-Cross Attention, conditioning on clean source latents while denoising noisy sparse targets. Inference (right): A single trained model supports three inference modes: (1) recovering camera trajectories from video; (2) camera-controlled video generation given a target trajectory; and (3) joint generation of a synchronized video and camera trajectory from sparse input images.

Panel of Figure 3 — Figure 3: Qualitative Results on DL3DV. We visualize the NVS performance of our model against the state-of-the-art baseline Kaleido ( Liu et al., 2026 ) on DL3DV. Given a single reference image (first column), our model synthesizes target views (third column) that exhibit superior structural fidelity and lighting consistency, closely matching the ground truth (fourth column). Note that ours predicts the camera parameters by itself.

Representation	Formulation	Dimension	Limitation
Camera Matrices	$K,R,T$	$(4,4)$	spatially unaligned
Plücker Embedding	$[R\mathbf{d},R\mathbf{d}\times T]$	$(H,W,6)$	input channel mismatch
Raymap	$[T,R\mathbf{d}]$	$(H,W,6)$	input channel mismatch
Pointmap	$RX\cdot D+T$	$(H,W,3)$	depth required
Raxels (Ours)	$R\mathbf{d}+T$	$(H,W,3)$	—

Table 1: Comparison of Camera Representations. We define the ray direction

\mathbf{d}=X/\|X\|

, where

X

denotes the unprojected pixel coordinates obtained from the intrinsic matrix

K

R

and

T

denote rotation and translation,

D

denotes depth, and

\times

denotes the cross product. Raxels are the only representation that matches both the spatial structure and the 3-channel input format of a pretrained video diffusion VAE.

Figure 4: Decoupled Self-Cross Attention. We replace standard self-attention in the video diffusion backbone with a Decoupled Self-Cross Attention block that processes video and ray latents in parallel branches. Within each branch, intra-modal self-attention operates on tokens of the same modality, followed by inter-modal cross-attention where queries from one modality attend to keys and values from the other. This separation encourages stable training and allows the video and ray latents to attend to each other.

Panel of Figure 5 — Figure 5: Qualitative Results on DL3DV-140 following the predefined camera trajectory. We visualize pose-conditioned video generation given a single input image from DL3DV-140 ( Ling et al., 2024 ) and a specific camera path. These scenes present distinct challenges: the top example features a cluttered layout with an off-center subject, while the bottom example contains highly reflective metallic surfaces. Top: The model renders the scene from “Arc Left” camera movement while maintaining the geometric consistency of sofa and many objects within a frame. Bottom: The model is capable of synthesizing the scene from “Arc Right” trajectory, notably our model can synthesize view-dependent reflections on the metallic surfaces while following the camera path.

Panel of Figure 6 — Figure 6: Self-Consistency and Ablation Study. Re-generated frames from the cycle self-consistency test on three scenes from DL3DV. From left to right: with Plücker embeddings replacing raxels, without DSCA, without cosine similarity loss, our full model, and ground truth. Replacing raxels with Plücker embeddings causes the most severe degradation, consistent with the quantitative results in Table 2 .

Method	FID $\downarrow$	FVD $\downarrow$	$R_{\text{err}}\downarrow$	$T_{\text{err}}\downarrow$
Ours	7.33	68.17	0.020	0.018
w/o DSCA	8.69	77.08	0.048	0.052
w/o Cosine Sim. Loss	9.48	97.84	0.058	0.094
Plücker Embedding	21.97	333.56	0.241	0.430

Table 2: Self-Consistency and Ablation Study. We evaluate cycle consistency on DL3DV (sampled at 12 FPS from raw videos) with three source images. Each row shows the effect of removing or replacing one design choice. Replacing raxels with Plücker embeddings degrades all four metrics by large margins, while removing DSCA or the cosine similarity loss produces smaller but consistent drops. Lower is better for all metrics.

Dataset	1 step	2 steps	5 steps	20 steps	VGGT
Re10K	95.39	95.91	95.08	94.78	98.07
DL3DV-140	82.78	88.37	87.30	85.90	91.86
T&T	92.34	93.51	93.43	93.01	97.70

Table 3: Camera Pose Estimation. Mean Relative Rotation Accuracy (mRRA@30,

\times 100

) across diffusion step counts on three benchmarks, with VGGT ( Wang et al., 2025 ) as a reference. Our model reaches its best accuracy at 2 diffusion steps across all benchmarks. Higher is better.

Method	Metrics
Dataset	FID $\downarrow$	FVD $\downarrow$	$R_{\text{err}}\downarrow$	$T_{\text{err}}\downarrow$
RealEstate10K
MotionCtrl	22.58	229.34	0.231	0.794
VD3D	21.40	187.55	0.053	0.126
ViewCrafter	20.89	203.71	0.054	0.152
Wonderland	16.16	153.48	0.046	0.093

Table 4: Quantitative Comparison of Camera-Controlled Video Generation Evaluations. We evaluate visual quality (FID

\downarrow

, FVD

\downarrow

) and trajectory adherence (

R_{\text{err}}\downarrow

T_{\text{err}}\downarrow

) across three benchmarks, and achieve the best performance on visual qualities (FID, FVD) with temporal coherence on generated videos.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 142 papers

WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models
Oct 22, 2026Chenxi Song, Yanming Yang, Tong Zhao +3Camera-Controlled Video Generation Trajectory-Guided Video Generation
Westlake University, Hangzhou, China · Beihang University, Beijing, China · Singapore Management University, Singapore
PartiCam: Camera Controlled Video Generation with Reward Guidance
Sep 30, 2026Amine Ouasfi, Runjia Li, Junlin Han +3Diffusion Model Guidance Camera-Controlled Video Generation
INRIA, Univ. Rennes, CNRS, IRISA · University of Oxford · Meta
Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation
Jan 15, 2026Dong-Yu Chen, Yixin Guo, Shuojin Yang +2Camera-Controlled Video Generation Video Diffusion Models
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China

Oct 22, 2026J/K move · Enter open · S save

WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models

Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li+2

Westlake University, Hangzhou, China · Beihang University, Beijing, China · Singapore Management University, Singapore

Camera-Controlled Video Generation Trajectory-Guided Video Generation Video Diffusion Models Diffusion Model Guidance Diffusion Model Adaptation

Open paper AskPDF arXiv

Video diffusion models have learned rich 3D/4D priors, making them promising for scene generation and video re-rendering. However, their use in spatial tasks is limited by imprecise camera control, spatial and temporal inconsistencies, and entangled camera motion with scene dynamics. Existing approaches, such as task-specific fine-tuning and post-process warping, often introduce visual artifacts, generalize poorly, or incur high computational costs. We propose WorldForge V2 to achieve accurate camera control while preserving the generative capabilities of pretrained video models. We first develop a training-free guidance framework using warped references to iteratively refine generation along a target camera path. Optical-flow analysis identifies motion-related latent channels for selective guidance, helping separate motion control from appearance. A dual-path strategy further compares guided and unguided predictions to correct artifacts from imperfect references. Together, these components guide viewpoint changes while preserving scene appearance and object motion. To reduce refinement costs, v-cache approximates the correction using the preceding denoising prediction, avoiding an additional model evaluation. We further extend motion localization from latent channels to network blocks and introduce Trajectory LoRA, which concentrates adaptation on blocks most responsive to trajectory guidance. A two-stage training strategy teaches these adapters to denoise the states encountered during guided generation, using limited supervision effectively. This enables reliable camera control with very little training data, with a single training scene already supporting generalization to unseen scenes. Quantitative and qualitative evaluations across multiple datasets show that our method achieves state-of-the-art generation quality and trajectory accuracy in camera-controlled video generation.

Abstract

Camera-Controlled Video Generation

Explore similar work

WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models

PartiCam: Camera Controlled Video Generation with Reward Guidance

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation