Diffusion models have demonstrated high-quality performance in conditional text-to-image generation, particularly with structural cues such as edges, layouts, and depth. However, lighting conditions have received limited attention and remain difficult to control within the generative process. Existing methods handle lighting through a two-stage pipeline that relights images after generation, which is inefficient. Moreover, they rely on fine-tuning with large datasets and heavy computation, limiting their adaptability to new models and tasks. To address this, we propose a novel Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation (LGTM), which manipulates the initial latent noise of the diffusion process to guide image generation with text prompts and user-specified light directions. Through a channel-wise analysis of the latent space, we find that selectively manipulating latent channels enables fine-grained lighting control without fine-tuning or modifying the pre-trained model. Extensive experiments show that our method surpasses prompt-based baselines in lighting consistency, while preserving image quality and text alignment. This approach introduces new possibilities for dynamic, user-guided light control. Furthermore, it integrates seamlessly with models like ControlNet, demonstrating adaptability across diverse scenarios.
Figures & tables
Fig. 1: Existing prompt-engineering methods fail to generate differences in generated images with or without light-specific prompts, resulting in outputs that overlook specified lighting conditions. Our proposed LGTM effectively guides lighting during image generation, ensuring outputs align with text prompts and desired lighting directions without fine-tuning.
Fig. 2: Overview of our proposed LGTM. The user inputs a prompt p and a light condition l . The Light Conditional Generation module generates the light direction mask ml according to l for manipulating the initial noise in Stable Diffusion. Then, the vanilla Stable Diffusion model integrates these inputs, dynamically adjusting the latent space—particularly channel 1 —to reflect user-defined lighting conditions. Finally, it outputs the final image I .
Fig. 3: Channel-wise sensitivity analysis via scaling the initial latent noise. We scale a single latent channel zT(c) ( c∈{1,2,3,4} ) by a constant factor α , while keeping the prompt, random seed, and the other channels fixed. Scaling channel 1 consistently induces global brightness changes and alters the perceived illumination direction, whereas scaling channels 2–4 mainly affects chromatic attributes with limited impact on lighting.
Fig. 4: Qualitative Results. The existing model fails to control lighting conditions, often generating images with random or inconsistent lighting. In contrast, our approach effectively incorporates user-specified light direction and intensity, producing more natural and coherent lighting effects in the generated images.
Method
FID ↓
NIMA ↑
CLIP-I ↑
CLIP-T ↑
Left ↑
Right ↑
Cat
SDXL
69.57
5.44
0.671
0.317
52.8%
52.9%
Ours
79.13
5.66
0.663
0.320
79.0 %
77.3 %
Dog
SDXL
71.15
5.67
0.618
0.309
51.9%
52.3%
Ours
81.08
5.68
0.610
0.312
77.3 %
76.7 %
TABLE I: Quantitative results comparing ours with the baseline.
Method
FID ↓
NIMA ↑
CLIP-I ↑
CLIP-T ↑
Left ↑
Right ↑
Cat
ControlNet
71.56
5.46
0.664
0.296
51.2%
51.8%
Ours
83.63
5.66
0.668
0.315
77.3 %
76.2 %
Dog
ControlNet
76.63
5.56
0.606
0.283
51.6%
52.1%
Ours
83.54
5.62
0.613
0.307
77.7 %
73.2 %
TABLE II: Quantitative results with the ControlNet.
Fig. 5: Qualitative Results in Application. Existing models integrated with ControlNet [ 7 ] control edges but fail to handle lighting, often producing inconsistent results. Our approach combines user-specified light direction and intensity with edge control, generating images with natural lighting and precise structure, demonstrating versatility in handling multiple controls.
Fig. 6: Illustration of Light Alignment Behavior. LGTM aligns generated subjects with the specified light direction, even overriding explicit constraints from ControlNet (e.g., canny-edge-based facial orientation). This can enhance realism but may also result in unintended subject orientation.
We propose a fine-tuning method for flow-matching diffusion models aimed at realistic artificial light modeling without the need for a large training dataset. We address the task of controllable interior image editing, where the goal is to turn artificial light sources on or off while preserving the scene geometry, object placement, materials, and visual identity of the original image. To achieve this, we decompose the task into two independent formulations. We introduce the LoRA Direction Training Method, which extracts the pure direction of the LoRA adapter effect in the diffusion model flow field, and we also introduce specialized loss functions to ensure the realism of the inverse transformation. Additionally, the resulting increment map is used for more precise adjustment of the lighting color and temperature.
We present LightMover, a framework for controllable light manipulation in single images that leverages video diffusion priors to produce physically plausible illumination changes without re-rendering the scene. We formulate light editing as a sequence-to-sequence prediction problem in visual token space: given an image and light-control tokens, the model adjusts light position, color, and intensity together with resulting reflections, shadows, and falloff from a single view. This unified treatment of spatial (movement) and appearance (color, intensity) controls improves both manipulation and illumination understanding. We further introduce an adaptive token-pruning mechanism that preserves spatially informative tokens while compactly encoding non-spatial attributes, reducing control sequence length by 41% while maintaining editing fidelity. To train our framework, we construct a scalable rendering pipeline that generates large numbers of image pairs across varied light positions, colors, and intensities while keeping the scene content consistent with the original image. LightMover enables precise, independent control over light position, color, and intensity, and achieves high PSNR and strong semantic consistency (DINO, CLIP) across different tasks.
Gengze Zhou, Tianyu Wang, Soo Ye Kim +7
AIML, Adelaide University · Adobe Research · University of Hong Kong +1
Controlling illumination in images is essential for photography and visual content creation. While closed-source models have demonstrated impressive illumination control, open-source alternatives either require heavy control inputs like depth maps or do not release their data and code. We present a fully open-source and reproducible pipeline for learning illumination control in diffusion models. Our approach builds a data engine that transforms well-lit images into supervised training triplets consisting of a poorly-illuminated input image, a natural language lighting instruction, and a well-illuminated output image. We finetune a diffusion model on this data and demonstrate significant improvements over baseline SD 1.5, SDXL, and FLUX.1-dev models in perceptual similarity, structural similarity, and identity preservation. Our work provides a reproducible solution built entirely with open-source tools and publicly available data. We release all our code, data, and model weights publicly.