Recent advances in video synthesis have enabled realistic integration of real individuals, driving demand for identity-aware generation. While emerging methods support joint appearance and voice injection in audio-visual models, they primarily focus on single-subject settings. Multimodal identity integration across multiple subjects remains limited, and precise alignment between visual and vocal identities in multi-subject scenarios remains underexplored. We present Identity-as-Presence, a unified framework for joint personalized audio-video generation. An automated data curation pipeline constructs identity-labeled audio-visual pairs for single- and multi-subject scenes. A unified identity injection mechanism then binds paired appearance and voice through shared cross-modal identity binding and subject-anchored captions. A multi-stage training strategy further leverages large-scale unimodal data alongside scarce paired clips to mitigate modality imbalance. Experiments show superior audio quality, video fidelity, and audio-visual consistency, with stronger multi-subject binding than the compared methods. For more details and qualitative results, please refer to our webpage: Identity-as-Presence.
Figures & tables
Figure 2: Overview of the data curation pipeline: A. extracting paired appearance-voice identity signals from raw videos, B. matching cross-clip references for identity injection, and C. synthesizing structured captions with subject anchors.
Figure 3: The overall dual-tower DiT architecture and training framework. The model encodes video, audio, paired appearance/voice references, and subject-anchored captions, binds them with shared identity embeddings, and injects them via reference positioning and decoupled asymmetric attention. Training proceeds in three stages: unimodal identity, joint multimodal alignment, and spatio-temporal multi-view fine-tuning.
Method
English
Chinese
WER% ↓
AID-SIM ↑
WER% ↓
AID-SIM ↑
E2 TTS ( Eskimez et al. 2024 )
2.19
0.710
1.97
0.730
F5 TTS ( Chen et al. 2025b )
1.83
0.647
1.56
0.741
CosyVoice2 ( Du et al. 2024 )
2.57
0.652
1.45
0.748
CosyVoice3 ( Du et al. 2025 )
2.21
0.720
1.12
0.781
Ours-LTX-2.3 Audio
1.57
0.770
1.20
0.785
Table 1: Comparison with SOTA TTS Methods on Seed-TTS-Eval ( Anastassiou et al. 2024 ) .
Models
Audio
Video
A/V Consistency
Binding †
PQ ↑
CLAP ↑
FD ↓
WER ↓
AID-SIM ↑
AES ↑
DD ↑
OC ↑
VID-SIM ↑
Sync-C ↑
Sync-D ↓
ImageBind ↑
FM ↑
SM ↑
Phantom ‡
/
/
/
/
/
0.54/0.53
0.55/0.57
0.11 /0.09
0.74/0.49
/
/
/
0.62
/
CV3+HunyuanCustom
7.32
0.14
1.00
22.94
0.48
0.54
0.60
0.11
0.64
5.41
8.66
0.27
/
/
CV3+HuMo
7.35
0.10
1.06
24.09
0.47
0.54
0.33
0.11
0.71
5.35
8.86
0.28
/
/
QIE+UniAVGen
6.15
0.24
0.97
68.86
0.32
0.35
0.65
0.11
0.37
3.99
11.66
0.29
/
/
QIE+ID-LoRA
7.03
0.11
0.84
30.59
0.54
0.56
0.58
0.11
0.41
6.30
7.85
0.36
/
/
Table 2: Comparison under a matched identity-conditioned protocol. Bold and underline mark best and second-best over all reported scores for each metric (single and multi ranked separately).
Figure 4: Qualitative comparison with identity-aware video and joint audio-video baselines.
Figure A: Three appearance-voice binding variants aligned with the main-paper ablation. (A) Shared Identity Embeddings (IE) assign the same learnable embedding to each subject’s visual and audio references. (B) RoPE Offset (Entity RoPE buckets) places different subjects in separate temporal RoPE ranges. (C) Rotary Bias (Group RoPE) retains shared temporal positions and distinguishes subjects with group-specific rotary biases. Green denotes noisy target tokens. Red and blue denote references from two subjects.
Figure E: Qualitative results of single-subject personalized joint audio-video generation.
Figure F: Responding to audio descriptions in the prompt, the model jointly synthesizes identity-preserving speech and ambient sounds that stay consistent with the video.
Figure G: Qualitative results of multi-subject personalized joint audio-video generation.
Figure H: Failure cases: large head pose, heavy facial occlusion, extreme expression/pose, and noisy reference audio.