Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.
Figures & tables
Figure 1: Unified Audio-Visual Personalization with ID-LoRA . ID-LoRA takes a reference audio clip and a target first frame as input. Unlike cascaded pipelines that treat modalities separately, ID-LoRA jointly generates synchronized video and audio in a single pass. This unified approach allows a text prompt to simultaneously dictate novel content, such as a specific speaking style and environmental acoustics ( e.g. , “a jackhammer is drilling in the background”), while ensuring the subject’s vocal identity and visual likeness are preserved across the generated sequence. Video examples with audio are available in the supplementary material.
Figure 2: ID-LoRA (overview). Target first frame and reference audio are encoded into latents, concatenated with noisy targets, and fed to a shared LTX-2 DiT adapted with In-Context LoRA. Reference audio tokens receive negative temporal positions in the RoPE space, cleanly separating them from the target tokens. Joint text conditioning yields synchronized, identity-preserving video and audio whose acoustics follow the prompt and adapted to the new environment.
Method
Spk Sim ↑
Face Sim ↑
LSE-D ↓
LSE-C ↑
CLAP ↑
WER ↓
C-Easy
CosyVoice 3.0 + WAN2.2
0.510
0.886
11.27
1.53
0.301
0.273
VoiceCraft + WAN2.2
0.412
0.879
10.99
1.48
0.335
0.241
ElevenLabs + WAN2.2
0.413
0.879
11.98
1.51
0.266
0.180
Kling 2.6 Pro †
0.487
0.847
10.24
3.01
0.327
0.046
ID-LoRA (Ours)
0.573
0.870
8.20
4.43
0.372
0.106
C-Hard
CosyVoice 3.0 + WAN2.2
0.391
0.890
11.40
1.50
0.249
0.362
Table 1: Comparison on CelebV-HQ easy (same-video), hard (cross-video), and TalkVid splits. ↑ higher is better, ↓ lower is better. Best in bold , second best underlined . † Commercial model.
Method
Spk Sim ↑
Face Sim ↑
LSE-D ↓
LSE-C ↑
CLAP ↑
WER ↓
C-Easy
CosyVoice 3.0 + LTX
0.522
0.875
9.42
2.81
0.307
0.287
VoiceCraft + LTX
0.412
0.885
9.19
2.32
0.335
0.241
ElevenLabs + LTX
0.420
0.873
9.44
2.99
0.262
0.185
LTX-Zeroshot
0.462
0.881
8.05
2.41
0.308
4.634
ID-LoRA
0.573
0.870
8.20
4.43
0.372
0.106
C-Hard
CosyVoice 3.0 + LTX
0.409
0.888
9.39
2.13
0.246
0.362
Table 2: Ours vs LTX variants across three datasets: CelebV-HQ easy (same-video), CelebV-HQ hard (cross-video), and TalkVid. Includes TTS-conditioned LTX and LTX-Zeroshot variants. ↑ higher is better, ↓ lower is better. Best in bold , second best underlined .
Method
Spk Sim ↑
Face Sim ↑
LSE-D ↓
CLAP ↑
WER ↓
No Identity Guidance
0.438
0.875
8.71
0.364
0.113
Standard PEs
0.441
0.872
9.02
0.359
0.252
OURS (Neg. PEs + IdG)
0.477
0.874
8.50
0.363
0.113
Table 3: Core design ablation on the CelebV-HQ hard split (cross-video): no identity guidance, standard positional encoding, and full OURS (negative positions + identity guidance). The full setup gives the best overall balance on synchronization and intelligibility while retaining strong identity.
Figure 3: Human evaluation results: A/B preference rates (%) on the hard (cross-video) split. Annotators on AMT evaluated 35 pairs across 8 speakers along three axes: voice similarity, environment sounds, and speech manners.
Figure 4: Examples from the MOS evaluation set. Each column shows two generated videos from the same speaker under different environmental sound conditions.
Figure 5: Environment sound interaction MOS study: per-scenario mean opinion scores (1–5) with 95% confidence intervals for ID-LoRA and Kling 2.6 Pro across ten physical interaction scenarios. ID-LoRA scores higher on 8 of 10 scenarios and in overall score.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
CelebVHQ-Hard
CelebVHQ-Easy
TalkVid
Method
LipMot. ↑
FaceSim ↑
LipMot. ↑
FaceSim ↑
LipMot. ↑
FaceSim ↑
LTX-Zeroshot
0.95
0.947
1.73
0.944
2.41
0.927
VoiceCraft-MFA-LTX
1.03
0.944
0.97
0.947
2.94
0.832
CosyVoice-LTX
1.07
0.944
1.90
0.940
2.95
0.828
ElevenLabs-LTX
1.29
0.943
1.29
0.942
2.82
0.829
Ours
1.56
0.936
2.98
0.939
3.07
0.832
Appendix
Table 4: Lip motion ( ×102 ) vs. face similarity (FaceSim) for the top 30% highest-FaceSim samples of each method. Higher lip motion indicates more active mouth articulation. Across all datasets, methods with less lip movement consistently achieve higher FaceSim, revealing a systematic metric bias against realistic speech animation.
sid
Spk Sim ↑
Face Sim ↑
LSE-D ↓
CLAP ↑
WER ↓
2
0.459
0.872
8.63
0.348
0.117
4 (default)
0.477
0.874
8.50
0.363
0.113
6
0.477
0.874
8.45
0.355
0.142
8
0.481
0.873
8.52
0.361
0.185
10
0.485
0.872
8.56
0.353
0.119
12
0.482
0.876
8.69
0.350
0.145
Appendix
Table 5: Identity-guidance scale sweep sid on the hard split (two-stage), with negative temporal positions enabled. Larger sid generally improves speaker similarity but introduces trade-offs in lip-sync and ASR metrics.
Figure 6: Instruction pages shown to Amazon Mechanical Turk annotators at the beginning of each human evaluation study.
Figure 7: Example evaluation cards from both human evaluation studies.