UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

Video identity replacement seeks to transfer the identities of one or more subjects while preserving the motion, expressions, and temporal structure of a driving video. Existing methods largely target single-person settings and often require task-specific structural controls, such as masks or pose representations, limiting their flexibility in general multimodal editing systems. Progress on multi-person replacement is further constrained by the scarcity of paired training data. We present Vorch-IR, a unified framework that supports single- and dual-person identity replacement, with optional background replacement, in a single model. Built on LTX2, Vorch-IR jointly conditions on a driving video, indexed reference images, and a textual editing instruction. The reference images need not match the pose, layout, or spatial configuration of the driving video: their roles as subject or background references are specified through the instruction. Dense visual conditions are fused through self-attention, while a vision-language context establishes semantic correspondence through cross-attention. We further develop an automatic data construction pipeline that synthesizes paired supervision for all four editing settings. Experiments using automatic metrics and pairwise human evaluation demonstrate strong identity preservation, motion fidelity, and temporal coherence across diverse scenarios. A temporal overlapping inference strategy additionally extends the short-clip model to minute-long generation without autoregressive continuation.

Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity paired videos: real footage cannot provide different identities performing exactly the same motion, leaving the task without paired supervision or frame-aligned ground truth. Existing methods therefore rely on same-identity masked reconstruction, which restricts supervision to predefined editable regions. To address this, we introduce an identity-expression decoupled synthesis pipeline that constructs expression-synchronized cross-identity video pairs from real footage. Expression-bearing facial regions are retained under small clip-consistent geometric perturbations, while the surrounding head is regenerated with a new identity as synthesized swapping input. These pairs form a cross-identity benchmark with frame-aligned real-world video as ground-truth targets, enabling full-reference evaluation of identity, expression, pose, reconstruction fidelity, and temporal stability. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, supporting both training (20,278 videos) and benchmarking (1,040 videos). With the cross-identity supervision enabled by this paired dataset, we propose DirectSwap, a mask-free video head swapping training paradigm. Under otherwise identical settings, this formulation outperforms head- and rectangle-masked same-identity reconstruction, particularly for bidirectional head-silhouette changes. At inference, driving-output divergence and emergent reference attention estimate the edit support, allowing unchanged non-head content to be restored from the driving video without external segmentation or additional training. On 1,040 clips from 104 unseen subjects, the resulting model demonstrates that the proposed paired supervision supports effective whole-head swapping.

Diffusion-based talking head generation has achieved remarkable visual quality, yet scaling it to long-term videos remains challenging. The widely adopted chunk-wise paradigm introduces two fundamental failures: (1) temporal-spatial misalignment between static identity references and dynamic audio streams, and (2) cascading identity drift propagated through self-generated continuity references across chunks. To address both issues, we propose AsymTalker, a novel diffusion-based talking head generation method comprising Temporal Reference Encoding (TRE) and Asymmetric Knowledge Distillation (AKD). First, TRE mitigates temporal-spatial misalignment by transforming the static identity image into a temporally coherent latent representation through encoding of a temporally replicated pseudo-video, without introducing additional parameters. Second, AKD resolves the inherent conditioning dilemma in chunk-wise training: using ground-truth references causes train-inference mismatch, while self-generated references entangle supervision with identity drift. Our asymmetric design circumvents this by anchoring the teacher model with ground-truth continuity references to provide drift-free, chunk-level supervision, thereby avoiding the teacher bottleneck. Meanwhile, the student model learns under inference-aligned conditions, conditioned only on self-generated references, and is trained via distribution matching to preserve identity over long horizons. Extensive experiments show AsymTalker achieves state-of-the-art results on HDTF and VFHQ. It guarantees high-fidelity, identity-consistent synthesis over 600-second videos and reaches a real-time inference speed of 66 FPS.