Expression-Diverse References for Identity-Preserving Video Generation

Tianwen Fu; Wenbin Teng; Gonglin Chen; Junyi Ouyang; Haolin Xiong; Yajie Zhao

cs.CVOct 8, 2026

Expression-Diverse References for Identity-Preserving Video Generation

Authors: Tianwen Fu, Wenbin Teng, Gonglin Chen, Junyi Ouyang, Haolin Xiong, Yajie Zhao

Organizations: Institute for Creative Technologies, University of Southern California

Abstract

Identity-preserving video generation aims to maintain a subject's identity while synthesizing realistic videos. Yet a single reference portrait captures the subject's appearance under only one facial configuration. As expressions change, facial appearance can vary in highly identity-specific ways, leaving the subject's appearance under unseen expressions underdetermined by the reference alone. This expression-dependent variation also complicates evaluation: similarity to a neutral reference may decrease under strong expressions even for real images of the same person. We investigate this limitation from both generation and evaluation perspectives. First, we quantify how face-recognition similarity varies with expression intensity using controlled photographs and MEAD videos. We then construct a compact yet expressive reference gallery that captures diverse expression-dependent facial configurations. Matching against this gallery provides a more robust measure of identity similarity under expressive motion. To further expose performance degradation with expression intensity, we report identity similarity separately for mild, intense, and extreme expressions. For generation, we extend Stand-In to condition on our expression-diverse reference sets and develop a data-curation pipeline that extracts consistent yet diverse face crops from training videos. In practical settings where only a single portrait is available, we construct the reference set by synthesizing additional expressions with a pretrained facial reenactment model. On our controlled benchmark, both real and synthesized reference sets outperform the evaluated baselines in identity similarity across all three expression-intensity regimes, with the largest improvements for extreme expressions.

Figures & tables

Panel of Figure 1 — Figure 1: Expression-induced appearance changes are person-specific. Given the same expression instructions, the two subjects exhibit different changes in gaze, brow, and mouth geometry, illustrating information absent from a neutral reference.

Panel of Figure 2 — Figure 2: Example gallery for one of the 10 subjects with 8 expressions including neutral. All subjects were asked to perform the same set of expressions, whose instructions are listed in Appendix C . The gallery captures diverse expression-dependent facial configurations to support more robust identity evaluation and provide richer information for video generation models.

		ArcFace		CurricularFace
Dataset	$n$	Pearson’s $r$	Spearman’s $\rho$	Pearson’s $r$	Spearman’s $\rho$
Lightstage	70	$-0.587$	$-0.662$	$-0.636$	$-0.701$
MEAD	24,190	$-0.521$	$-0.590$	$-0.496$	$-0.567$

Table 1: Correlation between expression intensity

\Delta\Psi

and identity similarity on both datasets.

Panel of Figure 3 — Figure 3: ArcFace and CurricularFace similarity versus expression intensity on real images. Each point represents a non-neutral photograph (Lightstage) or video frame (MEAD), measured relative to the subject’s neutral reference. MEAD preprocessing is described in Appendix A.2 .

Figure 4: Distribution of expression intensity

\Delta\Psi

across the 70 non-neutral light-stage photographs. The solid curve is a kernel density estimate of the density histogram, and marks along the horizontal axis denote individual photographs.

Panel of Figure 5 — Figure 5: Representative facial artifacts produced by existing identity-preserving video generation methods under expressive motion. Each generated frame is shown beside the neutral reference portrait provided to the method.

Figure 6: Distribution of nearest-gallery CurricularFace identity similarity in different bins of expression intensity

\Delta\Psi

. We use all generated frames from the baselines and our method, run SMIRK ( Retsinas et al., 2024 ) to obtain the expression vector

\Psi

, and measure

\Delta\Psi=\|\Psi-\Psi_{\text{neutral}}\|_{2}

against the neutral reference (see Appendix A.1 for details). Each point gives the mean identity similarity within a unit-width

\Delta\Psi

bin, connected across bins; the final point aggregates the open-ended

\Delta\Psi\geq 12

bucket. Shaded regions denote 95% confidence intervals computed using clip-clustered standard errors. Dotted lines mark the thresholds between mild, intense, and extreme expressions.

Figure 7: Overview of our multi-reference subject-to-video generation pipeline.

Panel of Figure 8 — Figure 8: Qualitative comparison. The leftmost images are the subject’s neutral portrait and one expressive reference; each row shows four temporally ordered frames from the same generated video.

	All frames		Mild			Intense			Extreme
Model	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
Concat-ID [ICCVW2025]	59.31	58.19	24.6	71.64	70.44	54.7	58.47	57.17	20.7	46.82	46.30
Phantom [ICCV2025]	65.67	66.85	7.0	80.40	79.77	40.8	71.84	72.20	52.2	58.88	60.95
ConsisID [CVPR2025]	33.02	31.81	3.0	50.88	50.53	50.3	39.19	38.31	46.8	25.24	23.63
Stand-In [CVPR2026]	54.37	53.97	15.9	81.04	81.24	39.7	56.87	56.28	44.5	42.63	42.18
AnyID [CVPR2026]	62.83	62.74	7.8	73.86	73.77	54.4	66.36	66.02	37.8	55.48	55.77

Table 2: Identity fidelity on our evaluation dataset. The percentage columns report the fraction of each model’s frames in each expression-intensity regime.

	All pairs		Ours has higher peak intensity
Baseline	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
Concat-ID	97.9%	99.0%	96.1%	98.3%
Phantom	98.8%	98.5%	96.9%	96.9%
ConsisID	100.0%	100.0%	100.0%	100.0%
Stand-In	97.9%	98.3%	96.4%	96.4%
AnyID	99.0%	99.2%	96.9%	97.7%

Table 3: Win rates for Ours (Real) on matched subject–prompt pairs, using per-video mean nearest-gallery similarity. The right-hand columns include only pairs where ours has greater peak expression intensity.

		All frames		Mild			Intense			Extreme
Model	Refs.	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
AnyID	5	62.83	62.74	7.8	73.86	73.77	54.4	66.36	66.02	37.8	55.48	55.77
Ours	5	77.22	77.77	24.9	83.92	85.05	60.6	77.03	77.39	14.5	66.51	66.82
Ours	6	78.36	78.94	25.3	83.62	84.78	63.8	77.83	78.24	10.8	69.17	69.39
Ours	7	79.59	80.34	27.4	83.72	85.03	63.5	78.93	79.49	9.1	71.80	72.18
Ours	8	80.78	81.68	29.0	83.68	85.03	63.5	80.23	80.95	7.4	74.27	74.80

Table 4: Effect of reference count on identity fidelity. Each additional reference adds an expression to the preceding reference set; the ninth reference uses an held-out expression outside the standard gallery. Scores within 0.5 points of the best value in each similarity column are bolded.

	All frames		Mild			Intense			Extreme
Reference set	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
Repeated neutral	85.85	86.36	71.5	88.21	88.71	27.3	80.55	81.06	1.1	64.93	66.31
Pose-diverse neutral	57.27	57.41	8.7	67.75	67.71	60.0	59.46	59.52	31.2	50.11	50.47
Expression-diverse	80.78	81.68	29.0	83.68	85.03	63.5	80.23	80.95	7.4	74.27	74.80

Table 5: Effect of reference-image diversity on identity similarity. Scores within 0.5 points of the best value in each similarity column are bolded.

	All frames		Mild			Intense			Extreme
Temporal coordinates	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	%	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
Distinct negative	79.98	80.84	31.9	82.75	84.20	61.3	79.35	79.96	6.7	72.56	72.94
Shared $t=-1$ (ours)	80.78	81.68	29.0	83.68	85.03	63.5	80.23	80.95	7.4	74.27	74.80

Table 6: Effect of temporal positional embeddings for multiple references.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Held-out expression	ArcFace		CurricularFace
Held-out expression	Neutral	Nearest gallery	Neutral	Nearest gallery
H1	$78.05\pm 3.35$	$80.51\pm 3.05$	$78.70\pm 3.76$	$81.03\pm 3.32$
H2	$75.35\pm 3.53$	$80.94\pm 3.59$	$76.89\pm 3.43$	$82.33\pm 3.34$
H3	$77.85\pm 2.79$	$81.05\pm 2.50$	$79.96\pm 2.17$	$82.78\pm 2.32$
H4	$79.17\pm 2.80$	$81.21\pm 2.25$	$80.42\pm 2.30$	$82.02\pm 2.13$

Appendix

Table 7: Identity similarity of real photographs showing expressions excluded from the eight-image gallery (mean

\pm

standard error over 10 subjects).

	All frames		Mild		Intense		Extreme
Model	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$	$s_{\mathrm{Arc}}^{*}\uparrow$	$s_{\mathrm{Cur}}^{*}\uparrow$
Concat-ID	54.75	53.45	65.64	64.97	54.43	52.81	42.24	41.05
Phantom	61.77	62.11	73.01	72.99	66.77	66.96	56.20	56.71
ConsisID	30.41	29.72	47.02	46.89	36.45	35.91	22.90	22.02
Stand-In	51.87	51.50	74.53	75.33	54.02	53.34	41.28	40.74
AnyID	59.24	58.32	70.21	69.37	63.09	62.13	51.25	50.39

Appendix

Table 8: Nearest-gallery identity similarity using four held-out expressions. None of these photographs is used to condition any model. Mild, Intense, and Extreme restrict evaluation to frames in the corresponding expression-intensity regime.

Model	Face Similarity $\uparrow$
ConsisID	43.19
Concat-ID	50.05
Stand-In	47.99
Phantom-14B	55.04
Ours (Synthetic)	62.26

Appendix

Table 9: Official OpenS2V-Eval Face Similarity on the 60-video Human-Domain protocol (30 single-face and 30 single-human videos). Baseline scores are the published leaderboard values, and ours uses the same official evaluator and aggregation with references generated by LivePortrait.

Gallery expression	Verbal instruction
Neutral (0)	Eyes open, teeth together, and lips closed in a neutral resting expression.
Expression 1	Eyes squeezed shut with raised cheeks, a wrinkled nose, smiling lip corners, and slightly parted lips.
Expression 2	Eyes directed upward while the lips are pushed forward into a rounded funnel shape.
Expression 3	Brows drawn together near the center while the mouth stretches open wide.
Expression 4	Inner brows raised with one or both lip corners tightened backward into a restrained, dimpling expression.
Expression 5	Eyes looking up and to the left while the jaw shifts toward the right.

Appendix

Table 10: Verbal instructions associated with the neutral reference and expression images in the gallery.

Panel of Figure 9 — Figure 9: Example of a synthetic expression gallery generated from a single neutral portrait. Although less natural than the real captures in Figure 2 , the synthetic gallery provides additional expression information for identity-preserving video generation with the image-domain prior of the facial reenactment model.

Environment	Phrase	Anchor
kitchen	a warmly lit, everyday kitchen	stands at the counter
home office	a quiet, softly lit home office	sits at the desk
living room	a dimly lit, quiet living room	sits on the sofa
cafe	a quiet, softly daylit cafe	sits at a table by the window

Appendix

Table 11: Environment Phrases

Emotion	Cues
joy	eyes narrow into creases, cheeks lift high, the mouth opens wide showing teeth
surprise	brows arch high, eyes go wide with the whites showing, the jaw drops open
fear	brows raise and pull together, eyes stretch wide, the lips pull back flat and tense as the mouth opens
anger	brows pull down hard, eyes glare, lips press tight and then part to bare clenched teeth, nostrils flare
disgust	nose wrinkles, upper lip lifts, lip corners pull down, eyes squint
sadness	inner brows lift and knit together, lip corners pull down, the chin tightens and trembles, eyes fill with tears

Appendix

Table 12: Cues for each emotion

Explore similar work

Jan 4, 2026cs.CV

Slot-ID: Identity-Preserving Video Generation from Reference Videos via Slot-Based Temporal Identity Encoding

Human identity-preserving text-to-video generation remains challenging under large changes in viewpoint, facial expression, illumination, and motion. Existing methods condition the generator on a single reference portrait, but a static image cannot capture how identity-bearing cues evolve across views and expressions, leading to face deformation, pose locking, identity drift, or over-smoothed faces. We observe that a short reference clip naturally provides richer temporal and multi-view identity cues than any single image, motivating a video-referential formulation. This richer signal, however, introduces a new challenge: identity evidence is distributed across many frames and must be distilled into a compact, stable representation under a limited token budget. To this end, we propose Slot-ID, a lightweight identity-conditioning framework built on a frozen text-to-video backbone. Slot-ID employs a slot-based temporal identity encoder with Sinkhorn-routed iterative reading to distill a compact, stable set of identity tokens from the reference clip, complemented by an image-anchor stream for dual-source conditioning. Extensive experiments demonstrate that Slot-ID outperforms state-of-the-art methods in identity preservation and visual naturalness while remaining competitive in prompt following, with particularly large gains under challenging pose, expression, and motion variations.

Yixuan Lai, He Wang, Kun Zhou +1

State Key Lab of CAD&CG Zhejiang University Hangzhou, China · Department of Computer Science University College London London, UK

Video Diffusion Models Text-to-Video Generation Identity-Preserving Video Generation Video Generation

Ask PDF

Jun 11, 2026cs.CV

Avatar V: Scaling Video-Reference Avatar Video Generation

Generating avatar videos that are not merely visually similar to a target individual but behaviorally recognizable, faithfully reproducing their talking rhythm, gestural tendencies, and expression dynamics, remains an open challenge. Existing methods predominantly condition on single static images, which provide insufficient identity information and cannot capture dynamic motion traits, while standard pixel-level objectives underserve the perceptually critical facial regions that determine avatar fidelity. We present Avatar V, a production-scale framework that addresses these limitations through video-reference-conditioned identity modeling. Rather than compressing identity into fixed-size embeddings, the model conditions directly on the full token sequence of a reference video, learning to reproduce both static identity attributes (facial geometry, skin texture) and dynamic behavioral patterns (talking rhythm, micro-expressions) through attention over the reference context. We introduce Sparse Reference Attention, an asymmetric mechanism achieving linear-complexity conditioning on arbitrarily long references; a motion representation stream enabling closed-loop talking style transfer; and an identity-aware super-resolution refiner inheriting the full reference conditioning. These are supported by a data engine curating 100M+ training clips from 50M raw videos, and a five-stage training pipeline with flow matching pre-training, personality fine-tuning, two-phase distillation (>10x acceleration), and RLHF alignment, deployed across thousands of GPUs. Avatar V generates 1080p videos of unlimited duration, achieving state-of-the-art identity preservation, lip synchronization, and generation quality on our cross-scene benchmark, consistently outperforming leading systems including Seedance 2.0, Kling O3 Pro, Veo 3.1, and OmniHuman 1.5 in both automated metrics and human evaluation.

Benjamin Liang, Ce Chen, Desmond Lin +20

HeyGen Research

Video-Reference Video Generation Identity-Preserving Video Generation Talking Head Generation Human Avatar Animation Video Generation

Ask PDF

Jun 21, 2026cs.CV

Customizing Video Portraits via Identity-ActionDecoupling

Identity-Preserving Text-to-Video Generation (IPT2V) seeks to synthesize a temporally coherent video from a reference image and a textual description, while simultaneously preserving the subject's identity and allowing fine-grained control over facial dynamics. Although recent methods such as ID-Animator and ConsisID inject identity features only at inference time, they ignored the ID-irrelevant information contained in Facial embedding, leading to monotonous or inaccurate facial movements that poorly follow the prompt. We introduce Identity-Action Decoupling (IaD) framework as well as two loss function Identity Decoupling Loss and Text Alignment Loss to solve this problem. Without any subject-specific fine-tuning, IaD yields videos that (1) maintain cross-temporal identity consistency and (2) exhibit rich, controllable expressions and scene variations that closely match the input text.

Junxiong Lin, Haoran Wang, Xinji Mai +4

College of Intelligent Robotics and Advanced Manufacturing, Fudan University · The University of Hong Kong · College of Computer Science and Artificial Intelligence, Fudan University

Text-to-Video Generation Identity-Preserving Video Generation Video Generation Controllable Video Generation Temporal Consistency in Video Generation

Ask PDF