Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed. However, existing generation models often entangle these factors, limiting independent control. In this work, we introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions. This compositional approach enables flexible video generation by allowing arbitrary pairings of camera and content conditions, unlocking unprecedented creative control. To overcome the fundamental challenges of modality conflict and data scarcity inherent in such a system, we present two key innovations. First, we construct OmniCAM, a novel hybrid dataset combining curated real-world videos with synthetic data that provides diverse paired examples for robust multi-task learning. Second, we propose a Dual-level Curriculum Co-Training strategy that mitigates modality interference and synergistically learns from diverse data sources. This strategy operates on two levels: first, it progressively introduces control modalities by difficulties (condition-level), and second, trains for precise control on synthetic data before adapting to real data for photorealism (data-level). As a result, OmniCamera achieves state-of-the-art performance, enabling flexible control for complex camera movements while maintaining superior visual quality.
Figures & tables
Figure 2: Compositional camera control with multi-modal conditions. OmniCamera enables flexible combinations of multiple conditioning inputs, such as overlaying text-guided camera motion on top of trajectory or reference-video control, efficiently extending and diversifying camera motion effects.
Table 2
Figure 3: Pipeline of OmniCamera. Left: Diffusion Transformer with decoupled condition injection: text and reference video tokens are concatenated for self-attention; trajectory features are added to the noise latent. Right: Condition RoPE explicitly disentangles modalities using unique positional encodings.
Figure 4: OmniCAM dataset construction. Left: UE5 synthetic videos provide accurate camera poses to build paired and triplet supervision. Right: real videos are processed via trajectory estimation, trajectory filtering, trajectory classification, and intra-class matching to obtain reliable trajectories and cross-scene motion pairs.
Figure 5: Dual-level curriculum data composition. Condition-level curriculum progresses from text control (Stage I) to reference-video control (Stage II) to trajectory control (Stage III), following a coarse-to-fine and easy-to-hard order. In Stage II and Stage III, data-level curriculum first trains on synthetic data with precise trajectories (Substage 1), then fine-tunes on real-world videos to recover photorealism (Substage 2).
Figure 6: Qualitative results for text-controlled camera motion.
Table 7
Methods
T2V
I2V
V2V
FVD ↓
RotErr ↓
TransErr ↓
CLIP ↑
FVD ↓
RotErr ↓
TransErr ↓
CLIP ↑
FVD-V ↓
RotErr ↓
TransErr ↓
CLIP-V ↑
CamCloneMaster [ 26 ]
–
–
–
–
380.73
0.021
3.953
30.18
376.33
0.019
4.936
93.60
Ours
868.63
0.024
7.109
29.30
352.76
0.023
4.195
30.45
348.51
0.019
5.392
93.71
Table 5: Quantitative results of Reference-video-controlled camera motion on T2V, I2V, and V2V tasks. “-” means methods fail to accomplish task.
Figure 7: Qualitative results for reference-video-controlled camera motion.
Figure 8: Qualitative results for trajectory-controlled camera motion.
Figure 9: Visual comparison of OmniCamera with state-of-the-art methods. Text Control: Wan2.2-Fun-Camera produces some artifacts (e.g., the “ear” mutating into “eye”). Trajectory Control: ReCamMaster [ 2 ] produces severe distortions on the “bus” and “motorcyclist”. Reference-Video Control: CamCloneMaster [ 26 ] executes incorrect camera motion while introducing heavy background and object distortions.
Figure 10: Modality conflict analysis. Pairwise combination of contradictory camera conditions reveals a dominance order: trajectory>reference video>text prompts.
Method
Trajectory-controlled
Reference-video-controlled
T2V
I2V
V2V
T2V
I2V
V2V
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD-V ↓
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD-V ↓
w/o CC
3.512
0.052
1024.31
4.821
0.061
412.50
7.425
0.068
405.12
9.510
0.038
982.45
5.762
0.035
410.28
7.214
0.029
415.82
w/o DC
2.845
0.045
965.84
4.103
0.052
378.42
6.512
0.055
362.75
8.324
0.031
925.61
4.951
0.028
381.54
6.185
0.024
382.46
Ours
2.064
0.034
893.95
3.355
0.043
333.54
5.933
0.045
331.36
7.109
0.024
868.63
4.195
0.023
352.76
5.392
0.019
348.51
Table 6: Ablation study of Curriculum Co-Training across different tasks. CC : condition-level curriculum, DC : data-level curriculum.
Method
Trajectory-controlled
Reference-video-controlled
T2V
I2V
T2V
I2V
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD ↓
TransErr ↓
RotErr ↓
FVD ↓
w/o UE5
3.105
0.050
915.22
4.512
0.058
350.14
8.845
0.035
890.56
5.321
0.030
365.21
w/o Real
2.152
0.036
1150.45
3.481
0.045
485.62
7.315
0.026
1085.24
4.352
0.025
492.35
Ours
2.064
0.034
893.95
3.355
0.043
333.54
7.109
0.024
868.63
4.195
0.023
352.76
Table 7: Ablation study of Data Composition across different tasks. UE5 : UE5 synthetic data, Real : real-world data.
Figure 11: Qualitative ablation. (a) Removing DC leads to failed camera motions and degraded visual quality. (b)(c) Omitting CC results in structural distortions in I2V and content hallucinations in V2V (red boxes). (d) Training solely on synthetic data ensures precise control but lacks photorealism.
Camera motion control is essential for directing viewpoint changes in generative systems. However, existing methods typically condition the generation process on a single specific modality, such as explicit pose trajectories or reference videos, limiting their ability to support heterogeneous user inputs. To address this limitation, we present TriMotion, a modality-agnostic framework for camera-controlled video generation that maps video, pose, and text inputs, describing the same camera trajectory into a shared motion embedding space. Learning such a space requires synchronized supervision across modalities. Therefore, we build the Motion Triplet Dataset by extending a Multi-Cam Video Dataset with geometry-grounded motion descriptions derived from camera extrinsics. We further introduce a latent motion consistency objective that leverages the motion embedding space to encourage the generated video to follow the target camera trajectory directly in latent space, avoiding the cost of pixel-space decoding. Extensive experiments show that TriMotion generates high-quality videos that accurately follow the target camera trajectories across all three modalities. Beyond standard generation, the shared motion embedding space also enables flexible applications such as sequential motion composition and cross-modal motion interpolation.
Seunghyun Shin, Jifei Song, Wooseok Jeon +2
GIST · Huawei Noah’s Ark Lab · Yonsei University +1
We present Gen Through the Lens, a camera-controllable image-to-video generation framework that synthesizes videos from a given reference image while enabling realistic and physically consistent camera operations. Unlike previous models that treat camera motion as abstract conditioning or rely on a single coordinate system, our approach explicitly decomposes and fuses control signals across three coordinate spaces: the camera coordinate (modeled by intrinsic parameters), the world coordinate (represented by spatial geometry hints), and the object coordinate (described by 3D bounding boxes of target subjects). Through cross-coordinate conditioning fusion, these heterogeneous cues are unified to guide the generative process, enabling precise control of camera motion, focus, and framing while preserving temporal and spatial coherence. Extensive experiments demonstrate that our method produces controllable and visually consistent results across diverse camera operations, outperforming existing approaches in both realism and editability.
Controlling human motion and camera movement is essential for faithful human-oriented video generation, yet remains challenging in multi-person scenes with large body motions, occlusions, and dynamic cameras. Existing pipelines typically rely on visual motion sequences, such as skeleton maps, pose maps, or rendered body representations, for motion control, while using camera embeddings for camera control. Such heterogeneous control interfaces force video generation models to reconcile pixel-aligned visual cues with non-visual geometric embeddings, making motion-camera attribution difficult and sensitive to camera estimation errors. We propose UniMoCa, a representation-driven framework that unifies motion and camera controls in visual space. At the core of UniMoCa is Motion-Camera Visual Proxy (MCVP), a mutually-sharable novel representation that converts 3D human motion and camera trajectories extracted from driving videos into an identity-neutral visual proxy. MCVP renders temporally aligned human geometry under the recovered camera trajectory and augments it with explicit camera trajectory markers, replacing heterogeneous visual-parametric controls with distinguishable visual cues. As both control factors are represented in the same visual space, they become mutually compatible rather than heterogeneous, enabling consistent joint reasoning and editing during video generation. We further curate a MCVP-Video dataset covering complex actions, multi-person interactions, and diverse camera trajectories. Experiments based on the Wan2.2 I2V show that UniMoCa achieves substantial gains in human motion control, camera control, temporal consistency, and camera-aware robustness with minimal additional complexity. More details are shown in our Project page: https://tanliming-daniel.github.io/UniMoCa/.
Liming Tan, Ye Chen, Hao Zhang +3
Shanghai Jiao Tong University · Monash University · USC-SJTU Institute of Cultural and Creative Industry