Organizations: State Key Lab of CAD&CG, Zhejiang University, China · Cornell University, United States of America · 4DV.ai, China · Ant Group, China · Zhejiang University, China · Zhejiang Lab, China
Accurate lighting response is essential for realistic dynamic human relighting, because shadows, highlights, and color changes should follow the target illumination in both direction and relative strength. Current learning-based relighting methods can generate visually plausible videos, but they often exhibit imprecise lighting responses: shadows and highlights may drift away from the target direction, and the output brightness may not faithfully follow the relative intensities in the target environment, making the result closer to appearance harmonization than controllable relighting. To address this issue, we propose DynOLAT, a generalizable OLAT video prediction framework for dynamic full-body humans. Instead of directly synthesizing a single relit appearance, our method predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, and then linearly combines the predicted bases according to the target environment map. This OLAT-based formulation makes the directional response explicit in the predicted bases, while the relative lighting strength is determined by the linear weights derived from the environment map. We further introduce a dynamic capture pipeline and the first large-scale dynamic full-body OLAT dataset, comprising over 24 million frames from 224 identity-outfit combinations, where different outfits of the same person define separate combinations. Experiments show that DynOLAT responds more faithfully to variations in target lighting direction and relative intensity than the tested baselines, enabling controllable video relighting while preserving fine appearance details and temporal coherence. The dataset and processing tools will be publicly released to foster future research.
Figures & tables
Figure 1. Controllable dynamic full-body relighting from a single video. Each row compares a lighting reference (left), ours (middle), and a baseline (right). Top: ours better matches the direction and shadow orientation of a rotating point light than UniRelight ( He et al., 2025 ) . Bottom: ours follows relative environment-light intensity without the spurious red cast produced by DiffusionRenderer ( Liang et al., 2025 ) in the low-intensity red-chair region. Two-row comparison of lighting references, our results, and baseline results under rotating directional light and spatially varying environment illumination.
Figure 2. The overview of the relighting pipeline. Our model encodes a flat-lit video into latent features and uses Spherical Harmonics (SH) embeddings of the target light direction to modulate the generation. Through an iterative process, it transforms the flat-lit latents into target OLAT representations. These generated bases are then linearly combined to achieve high-fidelity relighting under any environment map. Pipeline diagram showing flat-lit video encoding, target-light spherical-harmonic conditioning, iterative OLAT prediction, and linear combination under an environment map.
Figure 3. Data processing pipeline for the Dynamic-subset . (1) Light Switch Sequence: One sequence adheres to a rigorous interleaved pattern: 18 flat-lit frames are paired with 9 frames of OLAT-A and 9 frames of OLAT-B, where A and B represent two distinct, spatially distant lighting directions. (2) OLAT to Flat-lit Warping: Within each sequence, we compute optical flow with AllTracker ( Harley et al., 2025 ) and foreground masks with BiRefNet ( Zheng et al., 2024 ) from the reference flat-lit frames. The OLAT frames are then warped to align with the flat-lit frames using the predicted flow to eliminate motion misalignment. (3) Training Pairs: Each sequence yields two temporally consistent video pairs. As shown, our light switching pattern successfully provides distinct illumination conditions for the identical motion sequence, ensuring robust disentanglement of light and geometry. For a detailed description of the processing pipeline, please refer to Sec. 4.3 . Three-stage data-processing diagram showing the interleaved light-switch sequence, optical-flow alignment of OLAT and flat-lit frames, and the resulting paired training videos.
Dataset
Type
Dynamic
#Frames
#Subj.
#Views
Res.
Dynamic OLAT ( Zhang et al., 2021 )
upper-body
✓
603 k
4
1
4 K
Goliath-4 ( Martinez et al., 2024 )
upper-body
✓
>1 M
4
110
4 K
NetFlix Data ( Mei et al., 2025 )
upper-body
✓
12 M
67
36
4 K
POLAR ( Chen et al., 2026 )
upper-body
✓
28M
220
32
4 K
Ultrastage ( Zhou et al., 2023 )
full-body
✗
192 k
100
32
8 K
HumanOLAT ( Teufel et al., 2025 )
full-body
✗
850 k
21
40
5 K
Table 1. Comparison of our OLAT dataset with existing OLAT datasets. To the best of our knowledge, this is the first dynamic full-body OLAT dataset. It contains 24M valid frames, compared with 0.85M frames in the previous largest full-body OLAT dataset ( Teufel et al., 2025 ) . Column definitions: Type : subject category (upper-body or full-body); Dynamic : presence of dynamic OLAT motion sequences; #Frames : total frame count across all views; #Subj. : number of identity-outfit combinations; #Views : number of cameras, including short-lens and telephoto arrays; Res. : approximate image resolution, where 1K ≈1920×1080 , 4K ≈3840×2160 , and 8K ≈7680×4320 .
Figure 4. Capture system overview. (a) Our LightStage. (b) The flat-lit illumination configuration. (c–d) Camera distributions for the Dynamic-subset (c) and the Static-subset (d), reconstructed using COLMAP ( Schönberger and Frahm, 2016 ) . Photographs of the LightStage and flat-lit configuration, followed by reconstructed camera layouts for the Dynamic- and Static-subsets.
Figure 5. Quality assessment. (a) Full-body view from the short-lens array. (b) High-resolution view from the telephoto array, capturing fine facial details. (c) Surface normals estimated via photometric stereo from the full OLAT stack, recovering pore-level geometry. (d) Image-based relighting result synthesized via linear superposition, illustrating the photometric consistency and dynamic range of our data. Full-body and telephoto captures, estimated surface normals, and an image-based relighting result from the captured OLAT stack.
Figure 6. Qualitative comparisons for video relighting. We present results on our OLAT dataset under various environment maps. Video-relighting comparisons among the input, target-light reference, our method, and baseline methods under several environment maps.
Figure 7. Qualitative comparisons for in-the-wild video relighting. We evaluate our method on (a) the Actor-HQ dataset ( Işık et al., 2023 ) and (b) unstructured mobile phone captures. Relighting outputs from our method and baselines for Actor-HQ and mobile-phone video inputs.
Figure 8. Qualitative comparisons for image relighting. Compared with the evaluated baselines, our result retains more of the input’s fine-grained structure and more closely matches the captured OLAT-based ground truth. The top row highlights the specular response on the face and the shading contrast along the subject’s rear arm. Image-relighting comparison with captured OLAT-based ground truth, emphasizing facial highlights, arm shading, and fine subject details.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
FLIP ↓
IC-Light ( Zhang et al., 2025a )
19.51
.8760
.1322
.1226
LBM ( Chadebec et al., 2025 )
23.08
.9185
.1002
.0874
UniRelight ( He et al., 2025 )
22.91
.9070
.0928
.0889
DiffusionRenderer ( Liang et al., 2025 )
23.11
.9057
.0900
.0841
Ours
26.23
.9169
.0567
.0602
Table 2. Quantitative comparison on image relighting. To ensure a fair assessment, we employ a channel-aligned evaluation strategy to mitigate the inherent scale ambiguity between lighting intensity and surface color. Under this protocol, our method better matches the captured OLAT-based relighting ground truth across most metrics, indicating improved light-response fidelity.
Methods
LPIPS (temp.) ↓
WE(RAFT) ↓
WE(alltracker) ↓
RelightVid ( Fang et al., 2025 )
.02763
.00107
.00112
UniRelight ( He et al., 2025 )
.02660
.00079
.00082
DiffusionRenderer ( Liang et al., 2025 )
.02629
.00078
.00080
Ours
.02532
.00065
.00060
Table 3. Quantitative comparison of temporal consistency.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
FLIP ↓
Cross Attn
24.94
.9364
.0981
.0696
Global AdaLN
26.80
.9473
.0780
.0596
Channel Concat
27.13
.9434
.0815
.0589
Frame Concat
27.67
.9481
.0750
.0556
Ours
27.61
.9514
.0727
.0554
Table 4. Quantitative comparison of ablated variants. We evaluate OLAT prediction under alternative input-frame and lighting-conditioning designs. The full model achieves the best overall performance across the reported perceptual and structural metrics.
Figure 9. Ablation study on input frames processing strategies. We compare our method against concatenation along the channel and frame dimensions. Results show that Latent Bridge Matching strategy excels at preserving input details. Comparison of channel concatenation, frame concatenation, and Latent Bridge Matching, with enlarged regions showing differences in retained details.
Figure 10. Ablation study on lighting conditioning mechanisms. We compare cross-attention, global AdaLN, and the proposed block-specific AdaLN strategy. Our result better matches the target OLAT image in the shown shadow placement and local details. Lighting-conditioning comparison among cross-attention, global AdaLN, and block-specific AdaLN against a target OLAT image.
Figure 11. Ablation study on strategies for length extrapolation: (a) position interpolation ( Chen et al., 2023 ) , (b) sliding iterative sampling ( Jin et al., 2025 ) , and (c) our recurrent bridge context. Results show that our method avoids the visible artifacts and lighting drift of the two alternatives. Frames from long-video inference using position interpolation, sliding iterative sampling, and recurrent bridge context, showing artifacts and lighting drift in the first two methods.
Figure 12. OLAT synthesis under rotating lighting directions. We visualize the predicted OLAT results for a specific video frame of an unseen subject under varying lighting directions. Note the physically plausible movement of cast shadows and the preservation of high-frequency clothing details as the light source rotates. One subject relit by a sequence of rotating target-light directions, with corresponding changes in cast-shadow position.
Figure 13. Temporal consistency in dynamic OLAT generation. Selected frames from a generated OLAT video sequence under a fixed directional light, sampled at 10-frame intervals. Our model maintains temporally coherent shading throughout the complex motion. Frames sampled at 10-frame intervals from a dynamic OLAT video under a fixed directional light.
Figure 14. Generalization to arbitrary viewpoints. We present OLAT predictions for the same subject captured from distinct camera views. For each view, the model successfully synthesizes the corresponding OLAT image under a specified lighting direction, demonstrating robust generalization across diverse observation viewpoints. Predicted OLAT images of the same subject from several captured camera viewpoints under one specified light direction.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Count
Distribution
Gender
Male
135
Female
89
Accessory / prop
None
118
Any accessory/prop
106
Appendix
Table 5. Combined dataset composition. All counts refer to the 224 identity-outfit combinations, not to unique participants. For accessories and props, None and Any accessory/prop are mutually exclusive; indented rows are multi-label breakdowns of Any accessory/prop and may overlap.
Figure 15. Analysis of rolling shutter artifacts. (a–c) Three consecutive frames contain an OLAT frame, a rolling-shutter (RS) frame, and a flat-lit frame. (d) An example of a "distinct" RS frame from the same sequence where the exposure boundary is clearly visible. Due to the vertical orientation of our sensors, the electronic rolling shutter scans horizontally (right-to-left). Ideally, this results in a sharp transition boundary as seen in (d). However, as shown in (b), high-frequency boundary lines are often smoothed out by video compression algorithms. Additionally, (a) illustrates a failure case for heuristic brightness-based detectors: direct opposing illumination causes pixel intensities to exceed those of the flat-lit frame, complicating simple thresholding methods. Four frames illustrating rolling-shutter detection: an OLAT frame, an artifact-contaminated frame, a flat-lit frame, and a frame with a distinct exposure boundary.
Figure 16. Evaluation of Color Correction. (a) and (b) show the raw and color-corrected frames of the calibration sequence, respectively. (c) and (d) demonstrate the generalization of the computed CCM to a human subject within the same view. A significant artifact is observed in the white sleeve region in (c) and (d). The CCM causes originally neutral white pixels (near saturation) to shift significantly towards a darker, tinted hue, violating the scene’s white balance. This severe deviation on neutral tones motivates our decision to keep CCMs optional. Calibration and human-subject frames before and after color correction, with the corrected white sleeve exhibiting a dark color cast.
Figure 17. Effect of the delighting module. The comparison illustrates that the module adapts in-the-wild inputs toward the flat-lit domain used by our OLAT predictor, making downstream relighting more consistent with the training setting. Comparison of an in-the-wild input, its delighted version, and downstream relighting results, showing reduced residual illumination after delighting.
Figure 18. Reconstruction limitations of the video VAE. We compare the ground-truth input frame (left) with the VAE reconstruction (right). The reconstruction exhibits a noticeable loss of high-frequency details, particularly in regions with rapid motion. Side-by-side comparison of an input frame and its video VAE reconstruction, with enlarged regions showing softened facial and fabric details.
Figure 24
Figure 22. Cross-dataset qualitative comparison on HumanOLAT ( Teufel et al., 2025 ) . All displayed method outputs are brightness-aligned to the corresponding ground truth for visualization. HumanOLAT ground-truth images and brightness-aligned outputs from several relighting methods shown side by side.
Figure 26
Figure 25. Examples of training pairs from the Dynamic-subset . This figure shows the details of video training pairs in one sequence from several views. Multi-view flat-lit and OLAT video-frame pairs from one dynamic capture sequence.
Figure 26. OLAT images from short-lens and telephoto cameras in the Static-subset . The left-up corner number represent lighting direction index. A grid of OLAT images from short-lens and telephoto cameras under several indexed lighting directions.
Figure 27. Snapshots from all views of an example subject in the Dynamic-subset . Snapshots of one dynamic capture shown from all synchronized camera viewpoints.
Figure 28. Snapshots from all views of an example subject in the Static-subset . Snapshots of one static capture shown from all synchronized camera viewpoints.
Figure 29. Overview of all identity-outfit combinations in the Dynamic-subset . Gallery of all identity-outfit combinations in the Dynamic-subset.
Figure 30. Overview of all identity-outfit combinations in the Static-subset . Gallery of all identity-outfit combinations in the Static-subset.