One Image Is Enough: Self-Generated Supervision for Personalizing Human Video Diffusion at Test Time

Sadman Rakib Pinon; Adamo Orsini; Yanis Falaki; Linlian Jiang; Jacob Kingi; Yang Wang; Xinxin Zuo

cs.CVNov 2, 2026

One Image Is Enough: Self-Generated Supervision for Personalizing Human Video Diffusion at Test Time

Authors: Sadman Rakib Pinon, Adamo Orsini, Yanis Falaki, Linlian Jiang, Jacob Kingi, Yang Wang, Xinxin Zuo

Organizations: Concordia University · Mila – Quebec AI Institute

Abstract

Pose-driven human video diffusion animates a person from a single reference image and a target pose sequence, yet it struggles to preserve identity, rendering the face and garments unfaithfully as the pose departs from it. Personalization is the standard remedy, but its supervision is missing: one image fixes identity at a single pose, while a multi-pose subject video cannot be assumed in practice. Instead, we synthesize the missing supervision. A domain-tuned, pose-conditioned generator produces pseudo-ground-truth frames at selected key poses. An inference-time reference-only sampler steers these candidates toward the subject and keeps the most identity-consistent. A per-subject low-rank adapter then trains on static clips of them, configured from the reference alone, with no subject video, ground truth, or labels. Although lower in fidelity, these synthetic frames supervise more effectively than real ones. Real frames as captured carry motion that erodes the temporal prior, while a static clip conveys appearance alone. On TikTok and UBC Fashion, the adapted model improves the base on every metric, inverts the identity-versus-motion trade-off, and matches or surpasses adaptation on five real frames despite using none. Code and weights are available at https://github.com/SadmanPinon/one-image-is-enough.

Figures & tables

Figure 2: Method overview. From one reference image $x_{\mathrm{ref}}$ and a driving pose sequence $\mathcal{P}$ , a pose-conditioned generator $G$ synthesizes pose-targeted pseudo-ground-truth frames $\mathcal{D}_{\mathrm{pseudo}}$ of the subject. (a) Reward-guided generation: for each key pose, $G$ draws several candidates as an interacting particle system and resamples them in-loop toward a DINO body-similarity reward to $x_{\mathrm{ref}}$ , then selects the final frame by ArcFace face identity. Every reward is read from the reference alone (our reference-only reward-guided sampler). (b) Test-time personalization: a per-subject LoRA $\Delta\theta$ adapts the frozen base animator $F_{\theta}$ on static clips of the selected frames. The adapted model $F_{\theta+\Delta\theta}$ renders the personalized video $\hat{V}$ . No subject video, no ground truth, and no human feedback are used.

Figure 3: Temporal stability. Top: the driving frames. For Base (middle) and Ours (bottom), the leftmost column is that method’s output frame and the remaining columns are consecutive frame-difference maps over the same frames. The non-personalized base produces widespread inter-frame residual across the face and torso (temporal flicker), whereas our adapted model confines residual to true motion boundaries and stays stable, reducing mean inter-frame residual by $60\%$ over the frames shown. Brighter indicates larger frame-to-frame change.

Figure 4: Qualitative comparison on hard poses. For two subjects we show the reference and driving pose alongside the base, our adapted model, and ground truth. Insets crop the face. The base reconstructs identity and garment detail inconsistently as the pose departs from the reference, while our adaptation preserves both.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	FID-VID $\downarrow$
MusePose [ 39 ]	18.20/22.20	0.757/0.896	0.248/0.067	41.99/14.95	532.75/96.17	14.60/10.94
Moore-AA [ 31 ]	18.62/20.83	0.764/0.880	0.230/0.093	37.28/27.74	501.22/149.66	12.40/10.13
ControlNeXt [ 33 ]	16.31/18.48	0.728/0.853	0.296/0.132	33.48/13.82	548.01/143.02	28.23/14.60
Animate-X [ 38 ]	16.71/22.15	0.743/0.893	0.285/0.069	32.77/10.11	508.63/70.47	17.47/11.87
MimicMotion [ 59 ]	19.30/23.80	0.751/0.913	0.220/0.061	34.88/15.40	472.51/80.89	9.30/8.17
RealisDance-DiT [ 62 ]	17.55/23.33	0.717/0.908	0.261/ 0.053	30.39/10.81	458.81/72.94	–/–

Table 1: Main comparison on TikTok / Fashion (each cell TikTok/Fashion). StableAnimator is quoted from its paper, which reports neither FID nor FID-VID and does not evaluate on UBC Fashion.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FaceID $\uparrow$	BodySIM $\uparrow$	FVD $\downarrow$	dtSSD $\downarrow$
MusePose [ 39 ]	20.25	0.767	0.217	0.498	0.877	131.1	295.2
Moore-AA [ 31 ]	18.67	0.690	0.356	0.402	0.820	229.3	435.9
ControlNeXt [ 33 ]	17.85	0.720	0.328	0.534	0.854	226.1	1077.1
Animate-X [ 38 ]	17.51	0.709	0.368	0.440	0.830	198.6	371.0
MimicMotion [ 59 ]	17.28	0.704	0.346	0.488	0.874	219.6	769.9
RealisDance-DiT [ 62 ]	18.43	0.663	0.339	0.574	0.885	181.5	594.4

Table 2: Quantitative comparison on cross-domain performance on the Unseen100-Val set after training on TikTok. Off the training distribution, our adaptation reaches the only FVD below

100

(

92.5

) and the lowest dtSSD, while every other method stays above

125

Panel of Figure 5 — Figure 5: Pseudo-frame count and supervision source. (a) FVD (orange) is lowest at $K{=}5$ ; FaceID (teal) dips at $K{=}9$ and peaks at $K{=}21$ at a similar FVD (values from Tab. 6 ). (b) Seven-metric comparison of supervision sources (outward $=$ better, per-axis min–max over the rows of Tab. 5 ): Ours forms nearly the outer envelope, while self-distillation collapses on body similarity and temporal consistency.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FaceID $\uparrow$ (long)	BodySIM $\uparrow$ (long)	FVD $\downarrow$	dtSSD $\downarrow$
Real-5 + recipe	20.81	0.847	0.208	0.615	0.888	157.3	375.1
Ours (full)	21.20	0.847	0.206	0.617	0.887	133.5	369.0
w/o selection	20.92	0.847	0.208	0.613	0.884	163.8	375.7
w/o static-clip	20.80	0.842	0.208	0.583	0.872	151.9	416.3
Generator $G$	18.74	0.806	0.266	0.640	0.864	$-$	$-$
UniAnimate-DiT (Base)	20.52	0.834	0.233	0.597	0.884	206.0	386.6

Table 4: Component ablation. Each designed component is removed and compared against real-frame supervision (Real-5) and the unadapted base. FaceID and BodySIM are over the full video. Generator

G

is a per-frame reference with no temporal model, so it lacks FVD/dtSSD and is not ranked.

Supervision source	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FaceID $\uparrow$ (long)	BodySIM $\uparrow$ (long)	FVD $\downarrow$	dtSSD $\downarrow$
UniAnimate-DiT (Base)	20.52	0.834	0.233	0.597	0.884	206.0	386.6
Self-distillation (base’s own frames)	20.02	0.831	0.235	0.605	0.869	249.4	394.1
Reference image only (no $G$ )	21.14	0.848	0.205	0.608	0.885	139.0	380.5
Real-5 frames	20.81	0.847	0.208	0.615	0.888	157.3	375.1
Ours ( $G$ pseudo-GT)	21.20	0.847	0.206	0.617	0.887	133.5	369.0

Table 5: Supervision source. All rows share the adaptation recipe; only the supervision source changes. FaceID and BodySIM are over the full video, on the TikTok dataset. The standalone quality of

G

appears in Tab. 4 .

$K$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FaceID $\uparrow$ (long)	BodySIM $\uparrow$ (long)	FVD $\downarrow$	FID $\downarrow$	dtSSD $\downarrow$
1	20.33	0.836	0.232	0.590	0.875	148.3	25.40	388.5
3	20.93	0.845	0.210	0.605	0.881	178.6	27.82	377.5
5 (ours)	21.20	0.847	0.206	0.617	0.887	133.5	26.74	369.0
9	20.79	0.842	0.212	0.600	0.872	152.6	28.46	396.0
21	21.16	0.846	0.209	0.626	0.881	134.0	27.64	377.7

Table 6: Number of pseudo frames

K

. We sweep the number of self-generated supervision frames used for per-subject adaptation. FaceID and BodySIM are over the full video;

K{=}5

is the best balanced configuration.

Steer (in-loop)	Select (final)	PSNR $\uparrow$	FaceID $\uparrow$	BodySIM $\uparrow$	FVD $\downarrow$
none (i.i.d.)	DINO	21.48	0.685	0.916	713
none (i.i.d.)	ArcFace	22.42	0.695	0.918	538
DINO	DINO	22.40	0.661	0.921	516
DINO	ArcFace	22.80	0.711	0.928	496
ArcFace	ArcFace	21.07	0.655	0.922	429

Table 7: Reward-design ablation (on a tuning subset). The two reference-only rewards assigned to in-loop steering versus final selection. Steering with DINO and selecting with ArcFace is best.

Figure 6: Pose-conditioned generator $G$ . $G$ is built on FLUX.1 Kontext [ 21 ] paired with a pose ControlNet [ 57 ] . The target pose is VAE-encoded and read by a frozen ControlNet ( $14$ joint blocks, $38$ unified blocks), whose pose-aware features condition the main DiT. The reference image and initial noise are encoded and denoised by the main DiT, into which we inject the trainable LoRA [ 13 ] ; a text encoder supplies the prompt. The output latent is VAE-decoded to the subject rendered under the target pose. Backbone weights are frozen (snowflake); only the LoRA adapters are trained (flame).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Human preference study. Distribution of 1–5 ratings for ours, Animate-X, the unadapted base, and Moore-AnimateAnyone, on four axes. Thirty-four respondents, $180$ ratings per bar group. Ours places $93\%$ of appearance-alignment ratings at 4–5, against $38\%$ for the base model; the ordering is the same on all four axes. Protocol in Sec. B.2 .

$r$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FaceID $\uparrow$	BodySIM $\uparrow$	FVD $\downarrow$	dtSSD $\downarrow$
$4$	20.12	0.820	0.232	0.558	0.858	176.4	410.2
$8$	20.61	0.829	0.224	0.581	0.866	161.2	397.5
$16$	20.94	0.837	0.216	0.597	0.872	150.1	384.0
$32$	21.08	0.841	0.211	0.606	0.877	143.6	379.1
$64$	21.31	0.846	0.206	0.622	0.883	136.2	363.9
$\mathbf{128}$	21.49	0.852	0.201	0.635	0.890	129.8	365.1

Appendix

Table 11: Per-subject LoRA rank

r

(on a tuning subset). Adaptation quality as the rank of the per-subject update varies. Quality improves with capacity up to

r{=}128

, our default; beyond it (

r{=}256

) the adapter over-fits the few pseudo-frames and quality regresses, marking a clear optimum. Absolute scores differ from the main tables because this sweep runs on a tuning subset.

Figure 8: In-Domain Comparisons: Diverse Motions and Backgrounds. Examples illustrating variations in background, motion complexity, and clothing style. Our approach consistently preserves holistic appearance and pose alignment while producing sharper, more realistic details than competing methods.

Figure 9: In-Domain Comparisons: Diverse Appearance and Illumination. Examples covering variations in skin tone, lighting conditions, and clothing texture. Our approach maintains consistent subject appearance and realistic fine-grained details across all scenarios, clearly outperforming state-of-the-art methods.

Figure 10: Cross-Domain Comparisons: Animated Characters. Examples from the Unseen100-Val evaluation set featuring 2D and 3D animated characters unseen during training. Our approach generalizes beyond photorealistic subjects, faithfully preserving stylized appearance details and character proportions while maintaining temporal coherence under complex motion.

Figure 11: Adaptation is bounded by pseudo-frame quality. Top (HQ): high-quality pseudo frames ( Supervision ) yield faithful adapted outputs ( Output frames ). Bottom (LQ): degraded pseudo frames for the same subject yield weaker outputs. This is why the reference-only sampler selects high-quality frames before adaptation.

Identity-Preserving Video Generation

Identity-preserving video generation keeps a person or character recognizable as appearance, pose, motion, and scenes change.

See all 68 papers

ViDS: Video Diffusion Shader using 3D Face Tracking
Jul 27, 2026Wenbo Ji, Davide Davoli, Zhe Chen +3Identity-Preserving Video Generation Video Generation
Technical University of Munich · Toyota Motor Europe NV/SA · Woven by Toyota
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
Apr 21, 2026Zhengwentai Sun, Keru Zheng, Chenghong Li +7Camera-Controlled Video Generation Video Generation
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Future Network of Intelligence Institute, CUHK-Shenzhen, China
Vera: Identity-Faithful Human Subject-to-Video Generation
Jul 22, 2026Yulong Xu, Xinyue Liu, Shujuan Li +6Video Diffusion Models Identity-Preserving Video Generation
Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences · Tsinghua University

Jul 27, 2026J/K move · Enter open · S save

ViDS: Video Diffusion Shader using 3D Face Tracking

Wenbo Ji, Davide Davoli, Zhe Chen, Liam Schoneveld+2

Technical University of Munich · Toyota Motor Europe NV/SA · Woven by Toyota

Identity-Preserving Video Generation Video Generation Pose-Conditioned Video Generation

Open paper AskPDF arXiv

We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/

Abstract

Identity-Preserving Video Generation

Explore similar work

ViDS: Video Diffusion Shader using 3D Face Tracking

ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis

Vera: Identity-Faithful Human Subject-to-Video Generation