Performance, the externalization of intent, emotion, and personality through visual, vocal, and temporal behavior, is what makes a character alive. Learning such performance from video is a promising alternative to traditional 3D pipelines. However, existing video models struggle to jointly achieve high expressiveness, real-time inference, and long-horizon identity stability, a tension we call the performance trilemma. Conversation is the most comprehensive performance scenario, as characters simultaneously speak, listen, react, and emote while maintaining identity over time. To address this, we present LPM 1.0 (Large Performance Model), focusing on single-person full-duplex audio-visual conversational performance. Concretely, we build a multimodal human-centric dataset through strict filtering, speaking-listening audio-video pairing, performance understanding, and identity-aware multi-reference extraction; train a 17B-parameter Diffusion Transformer (Base LPM) for highly controllable, identity-consistent performance through multimodal conditioning; and distill it into a causal streaming generator (Online LPM) for low-latency, infinite-length interaction. At inference, given a character image with identity-aware references, LPM 1.0 generates listening videos from user audio and speaking videos from synthesized audio, with text prompts for motion control, all at real-time speed with identity-stable, infinite-length generation. LPM 1.0 thus serves as a visual engine for conversational agents, live streaming characters, and game NPCs. To systematically evaluate this setting, we propose LPM-Bench, the first benchmark for interactive character performance. LPM 1.0 achieves state-of-the-art results across all evaluated dimensions while maintaining real-time inference.
Figures & tables
Figure 1: LPM 1.0 generates identity-consistent conversational video with synchronized verbal and non-verbal behaviors—speaking, listening, micro-expressions, and natural motion—while maintaining visual fidelity across streaming and long-horizon video generation.
Figure 2: Data filtering and classification pipeline across four stages. Raw video is progressively filtered through single-shot extraction, quality filtering and cropping, conversation detection and clipping, and finally captioning with embedding generation to produce high-diversity, semantically rich, and emotionally expressive trainable clips.
Figure 3: Illustration of the conversational audio-video data processing pipeline. (1) Tracking and cropping converts multi-person clips into single-person clips; (2) Three-state labeling applies fine-tuned LR-ASD to produce frame-wise speak/listen/idle states; (3) Refinement and audio separation verifies and filters labels, then outputs speaker/listener-only audio tracks for retained clips.
GT
Precision (%)
F1 score
Speak
Listen
Domain 1
Pred.
Speak
44,256 (91.62%)
6,148 (12.01%)
87.80
89.67
Listen
4,049 (8.38%)
45,055 (87.99%)
91.75
89.83
Total (GT)
48,305
51,203
Acc. = 89.75 Macro-F1 = 89.75
Domain 2
Table 1: Frame-level results for speak-versus-listen classification after re-ranking and idle merging on 2K manually annotated test clips (1K clips each from Domain 1 and Domain 2). The left block shows the confusion matrix: entries are frame counts with column-normalized percentages in parentheses (equivalent to per-class recall on the diagonal). The right block reports per-class precision and F1 score. Summary metrics appear in the bottom row of each domain. Pred. : predicted label; GT : ground truth; Acc. : accuracy.
Model
Conversation
L_dlg
L_nondlg
Silence
Speak
Overall
Gemini 2.5 Pro (baseline)
76.80
67.58
64.76
63.96
69.88
70.47
Qwen3-Omni (fine-tuned)
76.22
76.48
56.77
83.36
80.55
78.37
Δ
− 0.58
+ 8.90
− 7.99
+ 19.40
+ 10.67
+ 7.90
Table 2: F1 scores of the six-category semantic verification model. Gemini serves as the closed-source baseline; Qwen3-Omni is fine-tuned on domain-specific conversational data. Metrics cover the five primary classes (excluding unknown ). Δ denotes the absolute improvement of Qwen3-Omni over Gemini. Abbreviations: L_dlg = Listen_dialogue, L_nondlg = Listen_nondialogue.
Figure 4: An example of our proposed multi-granularity identity-aware reference images. For each subject, we extract three complementary reference types from raw videos: (i) a global appearance reference capturing overall identity and background context; (ii) multi-view body references covering one to four viewpoints to provide appearance evidence; (iii) a set of facial expression references spanning one to eight expressive states, enabling faithful reproduction of identity-specific details.
Figure 5: Base LPM architecture. Inputs (noise video, the first frame, identity-aware reference images, text, speak audio, and listen audio) are encoded by modality-specific encoders and injected into a stack of DiT blocks via self-attention (visual tokens) and cross-attention (text and audio embeddings). The output video latent is decoded by a VAE decoder to produce the generated video.
Figure 6: Online LPM architecture. The generator DiT accepts noise inputs, streaming control signals (text, speak/listen audio), and identity reference images conditioned on noisy-history KV caches to produce renoised latents. The refiner DiT then recovers the final clean video chunks conditioned on clean-history KV caches. Both stages use chunk-wise causal attention masks for autoregressive rollout.
Figure 7: Execution timeline of the online interactive video system, illustrated with a full-duplex dialogue example. The system progresses through warmup, idle, listening, and responding states while maintaining continuous video output. Audio conditions (listen/speak) are aligned with the streaming timeline, and the Audio2Audio module introduces bounded latency between user input and response. The highlighted region shows the overlapping three-stage pipeline (Generator, Refiner, VAE) operating on chunked inputs. The cache structure (fixed sink + sliding window) enables stable long-horizon generation. Text conditioning and KV-cache operations are omitted for clarity.
Figure 8: Human preference evaluation (G/S/B) of Base LPM (720P) versus Kling-Avatar-2, OmniHuman-1.5 and Wan2.1-I2V on LPM-Bench . Overall measures which video appears more like a real person. The four diagnostic dimensions evaluate pairwise preference along specific axes.
Figure 9: Qualitative comparison of Base LPM (720P) with Kling-Avatar-2 and OmniHuman-1.5. Each row shows sampled frames from videos generated with the same reference image, text, and audio. Our model produces more diverse gestures, expressive emotion, accurate lip synchronization, and consistent identity.
Figure 10: Absolute score distributions (1–5 Likert) of Base LPM (720P) on LPM-Bench across Speak, Listen, and Conversation. Legend shows per-dimension means.
Figure 11: Human preference evaluation (G/S/B) of Online LPM (480P) versus LiveAvatar and SoulX on LPM-Bench . Each video pair is evaluated by three independent raters. The Overall axis uses an arena-style realism judgment, while the remaining axes separately evaluate motion dynamics, identity consistency, text controllability, and audio-video synchronization. Online LPM is strongly preferred to LiveAvatar across all dimensions, and is preferred overall to SoulX despite SoulX’s advantage on identity consistency.
Figure 12: Qualitative comparison of Online LPM (480P) with SoulX and LiveAvatar. Each row shows sampled frames generated from the same reference image and audio input. SoulX tends to produce conservative motion with limited gesture and head-pose variation, while LiveAvatar exhibits structural instability, background inconsistency, and lip-sync errors. Online LPM better balances motion richness, identity preservation, and temporal coherence.
Figure 13: Human preference evaluation (G/S/B) of Base LPM (480P) and Online LPM (480P) across three performance scenarios: (a) Speak, (b) Listen, and (c) Conversation. The online model remains competitive in terms of motion dynamics and identity consistency, while text controllability and audio-video synchronization aspects are similar.
Figure 14: Ablation on emotion references of Base LPM. With all other inputs fixed, adding emotion reference images improves the preservation of fine-grained identity-related expression cues, including smiling style, dental appearance, and subtle facial micro-expressions. In contrast, without emotion references, these person-specific expressive details are less faithfully preserved.
Figure 15: Ablation study on view references of Base LPM. Given a (i) global reference image and (ii) multi-view identity reference images, the model generates videos in which the subject undergoes substantial orientation changes while preserving 3D-consistent identity and appearance. This effect is evident not only in coherent body and facial structure, but also in orientation-dependent appearance details, such as the clothing logo on the back, which remains consistent as the subject turns.
Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.
Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini2.5Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.
Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar +1
Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending unified audio-visual synthesis to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and the generated identity may drift over long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.
Quanyue Song, Yishan He, Yanbo Ding +4
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China