X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis
Authors: Sonali Godavarthy, Matthias Neuwirth-Trapp, Tim-Felix Faasch, Maarten Bieshaar, Michael Moeller, Kristof Van Laerhoven, Danda Pani Paudel
Organizations: University of Siegen, Siegen, Germany · Bosch Research, Hildesheim, Germany · ETH Zürich, Zürich, Switzerland · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria
Imaging factor disentanglement in text-to-image generation aims to independently control image acquisition properties such as types of camera lenses, sensor types, viewpoints, and domains to enable combinatorial generalization. This should let the model synthesize novel factor combinations unobserved in the training data, such as pairing a fisheye lens with an event sensor never observed in training data. Recent work, MULTI, introduced learnable, factor-specific embeddings to disentangle imaging factors, along with the Factor Alignment Accuracy (FAA) metric to evaluate disentanglement quality. We identify and address two independent limitations. First, MULTI's pixel-level reconstruction objective supervises the model only on observed imaging factor combinations, providing no direct training signal for novel combinations. We therefore propose X-MULTI, which uses a pretrained vision-language model (VLM) to supervise novel factor combinations synthesized during training. Second, we show the FAA metric exhibits severe cross-factor correlation leakage, misrepresenting true disentanglement quality. We therefore propose Improved-FAA (I-FAA), which employs factor-specific augmentation strategies to break these correlations and enables more rigorous evaluation. Experiments demonstrate that X-MULTI achieves improved factor alignment on novel combinations compared to MULTI. Moreover, we show that correlation leakage in FAA distorts the evaluation of true factor disentanglement and I-FAA reduces this leakage and therefore provides a more robust assessment of factor alignment.
Figures & tables
Figure 1: X-MULTI Architecture . Overview of our framework, which augments MULTI (components enclosed within the dark grey dashed box) with zero-shot VLM supervision to optimize factor embeddings via joint diffusion and VLM-based factor alignment loss.
Factor
Prompt
Sensor
Classify the SENSOR type of this image (0-4): 0 = RGB: normal color camera 1 = Thermal: grayscale images with bright/warm colors/white 2 = Gated: high contrast black and white, harsh lighting, foggy/smoky 3 = Event: motion changes, red/blue motion capture 4 = RGB Thermal (Fusion): Color images containing an “overlay” or “glow” of thermal data. Look for heat signatures (bright orange/yellow/white highlights) mapped onto real-world objects and textures. Return only JSON: {"sensor": <int>}
Lens
Classify the LENS type of this image (0-1): 0 = Normal: standard camera lens 1 = Fisheye: distorted edges, wide field of view Return only JSON: {"lens": <int>}
Viewpoint
Classify the VIEWPOINT of this image (0-4): 0 = Drone: camera placed on a flying drone 1 = Pole: camera mounted on a pole 2 = Front: camera faces forward 3 = Back: camera faces backward 4 = Side: camera placed on left/right (typically on car) Return only JSON: {"viewpoint": <int>}
Domain
Classify the DOMAIN of this image (0-2): 0 = Real: real-world images 1 = Simulation: rendered or simulated images 2 = Video Game: captured from video-game Return only JSON: {"domain": <int>}
Table 1: Factor-specific Prompts for VLM-based supervision . Our method uses structured prompts and JSON output constraints to independently query the zero-shot VLM for each imaging factor.
Augmentation Category & Type
Lens
Sensor
Domain
Viewpoint
Target Classes
normal, fisheye
rgb, thermal, rgb-thermal, event, gated
real, simulation, video-game
front, back, side, pole, drone
Geometric
Horizontal flip
✓
✓
✓
Mild
Vertical flip
✓
✓
✓
—
Random rotation & translation
Mild
Moderate
Moderate
Mild
Color / Appearance
Table 2: Data augmentations for I-FAA classifier training. Category-specific, non-overlapping image augmentations applied during I-FAA (ours) training.
Table 3: Qualitative comparison of novel combinations. Image generation with unseen factor combinations. Our method, X-MULTI shows better factor adherence.
Table 4: Qualitative comparison of novel factor combinations with ControlNets. Image-to-image generation using ControlNet guidance with altered factors, showing X-MULTI (ours) achieves best factor adherence.
Method
Image Generation Metrics
Improved-Factor Alignment Accuracy (I-FAA)
IS [ 34 ] ↑
CLIP Score [ 16 ] ↑
DS [ 46 ] ↑
Lens ↑
Sensor ↑
Domain ↑
View ↑
Avg. ↑
factor
context
average
full-prompt
DreamBooth [ 32 ]
3.68
23.96
23.35
23.66
24.67
0.55
0.78
0.24
0.48
0.27
0.44
SDXL Zeroshot [ 30 ]
3.11
22.31
25.59
23.95
27.80
0.59
0.72
0.25
0.29
0.29
0.39
MULTI [ 13 ]
3.28
24.20
23.43
23.82
27.32
0.65
0.63
0.29
0.64
0.34
0.47
X-MULTI (ours)
3.23
24.62
23.57
24.10
28.22
0.59
0.76
0.30
0.66
0.40
0.53
Table 5: Quantitative comparison on novel factor combinations. Performance metrics across methods with and without ControlNet structural guidance, with best values highlighted in bold. X-MULTI (ours) achieves best overall disentanglement.
Table 6: Comparison of attention maps for FAA and I-FAA. Grad-CAM visualizations showing that I-FAA (ours) learns more semantically meaningful attention regions.
Figure 2: Cramer’s V correlations. Pairwise correlations between predicted factors for FAA (left) and I-FAA (ours) (right), showing that I-FAA reduces cross-factor information leakage. High values show strong correlations.
Metric
Lens
Sensor
Domain
View
Avg.
FAA [ 13 ]
0.23
0.83
0.50
0.17
0.43
I-FAA
1.00
0.97
0.98
0.84
0.95
Table 7: FAA and I-FAA on ground-truth DF-RICO images. Comparison of FAA and I-FAA (ours) metric on ground-truth images.
Table 8: Factor prediction comparison. Per-factor classification outputs compared against ground-truth labels for ground truth images. I-FAA (ours) predicts factor correctly.
Figure 3: Confusion matrices of the zero-shot VLM for factor classification: (a) Lens, (b) Sensor, (c) Domain, and (d) Viewpoint. Rows correspond to ground-truth labels and columns to predicted labels.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Table S.1 : Qualitative comparison of existing combinations. X-MULTI (ours) faithfully reproduces all target sensor, lens, and domain modalities present in the training distribution.
Method
Image Generation Metrics
Improved-Factor Alignment Accuracy (I-FAA)
FID [ 17 ] ↓
IS [ 34 ] ↑
CLIP Score [ 16 ] ↑
DS [ 46 ] ↑
Lens ↑
Sensor ↑
Domain ↑
View ↑
Avg. ↑
factor
context
average
full-prompt
DreamBooth [ 32 ]
58.27
5.02
24.52
23.75
24.64
24.86
0.57
1.00
0.91
0.96
0.80
0.91
SDXL Zeroshot [ 30 ]
60.19
4.36
20.70
24.68
21.19
25.13
0.59
0.65
0.71
0.65
0.36
0.59
Inspiration Tree [ 39 ]
55.35
5.08
19.78
21.89
20.84
23.78
0.59
0.60
0.64
0.68
0.42
0.58
MULTI [ 13 ]
60.13
4.68
23.82
24.77
24.29
27.30
0.56
0.98
0.90
0.94
0.82
0.91
Appendix
Table S.2 : Quantitative comparison on existing factor combinations. Best in bold. X-MULTI (ours) achieves the highest I-FAA metric while maintaining highly competitive underlying image generation quality scores.
Recent text-to-image models produce high-quality images, yet text ambiguity hinders precise control when specific styles or objects are required. There have been a number of recent works dealing with learning and composing multiple objects and patterns. However, current work focuses almost entirely on image content, overlooking imaging factors such as camera lens, sensor types, imaging viewpoints, and scenes' domain characteristics. We introduce this new challenge as Imaging Factor Disentanglement and show limitations of current approaches in the regime. We, therefore, propose the new method Multi-factor disentanglement through Textual Inversion (MULTI). It consists of two stages: in the first stage, we learn general factors, and in the second stage, we extract dataset-specific ones. This setup enables the extension of existing datasets and novel factor combinations, thereby reducing distribution gaps. It further supports modifications of specific factors and image-to-image generation via ControlNets. The evaluation on our new DF-RICO benchmark demonstrates the effectiveness of MULTI and highlights the importance of Factor Disentanglement as a new direction of research.
Large Vision-Language Models (LVLMs) exhibit strong performance on single-image tasks. However, their performance degrades significantly when handling multi-image inputs. While this degradation has been observed in prior work, its nature remains poorly understood. We empirically observe visual elements from different images become entangled in the model's representations and responses. We refer to this phenomenon as cross-image information leakage. To address this issue, we propose FOCUS, a training-free and architecture-agnostic method. FOCUS masks all but one image with random noise, guiding the model to focus on the single clean image. This process is applied across the target images to obtain logits under partially masked contexts. These logits are aggregated and then refined using a noise-only reference input, which suppresses the leakage and yields more accurate outputs. FOCUS consistently improves performance on diverse multi-image benchmarks. We further show that FOCUS generalizes to video understanding, extending its applicability beyond static multi-image inputs. This demonstrates that FOCUS offers a general solution for enhancing multi-image reasoning without additional training or architectural modifications.
Yeji Park, Minyoung Lee, Sanghyuk Chun +1
Sogang University · Princeton University · NAVER AI Lab
Multi-image reasoning remains a significant challenge for vision-language models (VLMs). We investigate a previously overlooked phenomenon: during chain-of-thought (CoT) generation, the text-to-image (T2I) attention of reasoning VLMs exhibits diffuse "pulses": sporadic and unfocused attention patterns that fail to concentrate on task-relevant images. We further reveal a systematic positional bias in attention allocation across images. Motivated by these observations, we propose PulseFocus, a training-free, inference-time method that structures CoT reasoning into interleaved plan/focus blocks with soft attention gating. By forcing the model to explicitly plan which image to examine and then gating decode-time attention to the referenced image, PulseFocus sharpens attention focus and yields consistent improvements on multi-image benchmarks like BLINK benchmark (+3.7%) and MuirBench (+1.07%).
Chenjun Li
School of Electrical and Computer Engineering, Cornell University