Recent proprietary models such as Sora2 demonstrate promising progress in generating multi-shot videos conditioned on multiple reference characters. However, academic research on this problem remains limited. We study this task and identify a core challenge: when reference images exhibit highly similar appearances, the model often suffers from reference confusion, where semantically similar tokens degrade the model's ability to retrieve the correct context. To address this, we introduce PoCo (Position Embedding as a Context Controller), which incorporates position encoding as additional context control beyond semantic retrieval. By employing side information of tokens, PoCo enables precise token-level matching while preserving implicit semantic consistency modeling. Building on PoCo, we develop a multi-reference and multi-shot video generation model capable of reliably controlling characters with extremely similar visual traits. Extensive experiments demonstrate that PoCo improves cross-shot consistency and reference fidelity compared with various baselines.
Figures & tables
Figure 1: Comparison of different strategies for multi-reference, multi-shot video generation. (A) Independent single-shot reference-to-video generation produces each shot separately, leading to inconsistent backgrounds and appearance details across shots. (B) Joint multi-shot reference-to-video generation improves global coherence, but without explicit side information, the model may associate a shot with the wrong reference, causing identity confusion. (C) Our PoCo with SideInfo-RoPE enables accurate shot-reference association, yielding consistent identity and background across shots. Right: spatially averaged self-attention over concatenated reference and shot tokens. AttnScore(Shoti→Refj) denotes the mean attention from Shoti to Refj .
Figure 2: We propose a multi-reference, multi-shot video generation model conditioned on reference images and per-shot captions. (a) The overall architecture integrates reference images, shot captions, and latent video features through VAE and MultiShot-DiT blocks. Each block contains Hierarchical Cross-Attention (b) and Self-Attention with SideInfo-RoPE (c). (b) The hierarchical mask allows reference tokens to attend to all captions, while video tokens in each shot attend only to their corresponding text segment. (c) SideInfo-RoPE assigns reference-specific phase codes in the rotary embedding space, so that temporally aligned shots inherit the corresponding phase patterns. Colored planes denote active rotations, while gray planes denote unrotated ones.
Figure 3: Data pipeline for multi-reference multi-shot video generation. The pipeline transforms raw long videos into multi-shot training samples. It includes video processing (quality filtering, shot segmentation, watermark removal, caption generation) and reference construction (face detection, ID clustering, background removal, and seedream-enhanced reference synthesis). These steps ensure clean, consistent identities and high-quality supervision for video generation.
Figure 4: Qualitative effect of SideInfo-RoPE on shot-level identity grounding. We test two pairs of visually similar female and male characters using descriptions associated with two reference portraits ( @character1 , @character2 ). Without SideInfo-RoPE, the model often exhibits incorrect or ambiguous identity grounding, including both identity swaps and failure cases that do not clearly match either reference. With SideInfo-RoPE, the intended reference-shot correspondence is preserved more reliably. Colored boxes indicate grounding to the corresponding reference identity, while black boxes denote ambiguous or failed grounding.
Figure 5: Comparison with commercial reference-to-video methods under the same text prompts and identity references. Compared with Kling-1.6 and Vidu-Q2, PoCo achieves better cross-shot continuity and cross-view consistency, preserving identity, scene layout, lighting, and fine-grained appearance more faithfully.
Figure 6: Comparison of temporal channel selection for SideInfo-RoPE. Thigh introduces facial motion artifacts and weaker identity consistency, while Tlow yields smoother motion and better identity preservation.
Multi-shot video generation creates sequences of distinct shots that follow a shared story while preserving subjects, settings, and temporal continuity.
Conditioning a video generator on multiple images requires preserving appearance while associating each reference with its intended role. We present MSR (Multiple Subject Reference), a slot-aware conditioning scheme for LTX-based video generation. Each reference image is independently encoded as a static clip and represented by a separate latent-token group. A compact Fourier-feature multilayer perceptron adds a numeric slot embedding, while slot-dependent temporal offsets modify the group's rotary coordinates. The reference groups are prepended to noisy target tokens and serve as clean context during target-only flow-matching training. We implement this scheme through low-rank adaptation and release the resulting weights and inference workflows. Qualitative examples demonstrate compositions containing distinct characters and referenced environments in realistic and stylized scenes. Development observations suggest reduced reference confusion relative to an earlier continuous-reference baseline, while similar clothing, complex garments, and viewpoint changes remain challenging. We describe the conditioning mechanism, the retained training configuration, and the observed strengths and limitations of the released system. A supplementary audio-reference experiment adds voice conditioning while keeping the visual parameters frozen.
Multi-shot video generation requires maintaining a consistent appearance of recurring entities across shots while remaining faithful to shot-specific text prompts. Recent autoregressive methods reuse previously generated frames as memory. However, full-frame storage entangles persistent entity information with transient scene context, leading to irrelevant information leakage and high computational cost. We propose an entity-centric memory in the form of an entity-indexed bank of latent patches. We introduce sparse token conditioning compatible with pretrained models, restricting self-attention to entity-relevant tokens and reducing computational cost. To support this, we introduce a structured multi-shot script format. We additionally propose a budgeted memory update strategy to maintain a compact, evolving memory. Finally, we equip the entity representation with a noise-injection mechanism that enables fine-grained appearance control, preventing leakage of irrelevant information. Our method improves prompt adherence and efficiency while preserving subject consistency.
Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang +2
Max Planck Institute for Informatics, Germany and Adobe Research, UK · Adobe Research, UK · Adobe Research, USA
Achieving semantic alignment across diverse video generation conditions remains a significant challenge. Methods that rely on explicit structural guidance often enforce rigid spatial constraints that limit semantic flexibility, whereas models tailored for individual control types lack interoperability and adaptability. These design bottlenecks hinder progress toward flexible and efficient semantic video generation. To address this, we propose Video2LoRA, a scalable and generalizable framework for semantic-controlled video generation that conditions on a reference video. Video2LoRA employs a lightweight hypernetwork to predict personalized LoRA weights for each semantic input, which are combined with auxiliary matrices to form adaptive LoRA modules integrated into a frozen diffusion backbone. This design enables the model to generate videos consistent with the reference semantics while preserving key style and content variations, eliminating the need for any per-condition training. Notably, the final model weights less than 150MB, making it highly efficient for storage and deployment. Video2LoRA achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.