X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis

Sonali Godavarthy; Matthias Neuwirth-Trapp; Tim-Felix Faasch; Maarten Bieshaar; Michael Moeller; Kristof Van Laerhoven; Danda Pani Paudel

cs.CVAug 25, 2026

X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis

Authors: Sonali Godavarthy, Matthias Neuwirth-Trapp, Tim-Felix Faasch, Maarten Bieshaar, Michael Moeller, Kristof Van Laerhoven, Danda Pani Paudel

Organizations: University of Siegen, Siegen, Germany · Bosch Research, Hildesheim, Germany · ETH Zürich, Zürich, Switzerland · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

Abstract

Imaging factor disentanglement in text-to-image generation aims to independently control image acquisition properties such as types of camera lenses, sensor types, viewpoints, and domains to enable combinatorial generalization. This should let the model synthesize novel factor combinations unobserved in the training data, such as pairing a fisheye lens with an event sensor never observed in training data. Recent work, MULTI, introduced learnable, factor-specific embeddings to disentangle imaging factors, along with the Factor Alignment Accuracy (FAA) metric to evaluate disentanglement quality. We identify and address two independent limitations. First, MULTI's pixel-level reconstruction objective supervises the model only on observed imaging factor combinations, providing no direct training signal for novel combinations. We therefore propose X-MULTI, which uses a pretrained vision-language model (VLM) to supervise novel factor combinations synthesized during training. Second, we show the FAA metric exhibits severe cross-factor correlation leakage, misrepresenting true disentanglement quality. We therefore propose Improved-FAA (I-FAA), which employs factor-specific augmentation strategies to break these correlations and enables more rigorous evaluation. Experiments demonstrate that X-MULTI achieves improved factor alignment on novel combinations compared to MULTI. Moreover, we show that correlation leakage in FAA distorts the evaluation of true factor disentanglement and I-FAA reduces this leakage and therefore provides a more robust assessment of factor alignment.

Figures & tables

Figure 1: X-MULTI Architecture . Overview of our framework, which augments MULTI (components enclosed within the dark grey dashed box) with zero-shot VLM supervision to optimize factor embeddings via joint diffusion and VLM-based factor alignment loss.

Factor	Prompt
Sensor	Classify the SENSOR type of this image (0-4): 0 = RGB: normal color camera 1 = Thermal: grayscale images with bright/warm colors/white 2 = Gated: high contrast black and white, harsh lighting, foggy/smoky 3 = Event: motion changes, red/blue motion capture 4 = RGB Thermal (Fusion): Color images containing an “overlay” or “glow” of thermal data. Look for heat signatures (bright orange/yellow/white highlights) mapped onto real-world objects and textures. Return only JSON: {"sensor": <int>}
Lens	Classify the LENS type of this image (0-1): 0 = Normal: standard camera lens 1 = Fisheye: distorted edges, wide field of view Return only JSON: {"lens": <int>}
Viewpoint	Classify the VIEWPOINT of this image (0-4): 0 = Drone: camera placed on a flying drone 1 = Pole: camera mounted on a pole 2 = Front: camera faces forward 3 = Back: camera faces backward 4 = Side: camera placed on left/right (typically on car) Return only JSON: {"viewpoint": <int>}
Domain	Classify the DOMAIN of this image (0-2): 0 = Real: real-world images 1 = Simulation: rendered or simulated images 2 = Video Game: captured from video-game Return only JSON: {"domain": <int>}

Table 1: Factor-specific Prompts for VLM-based supervision . Our method uses structured prompts and JSON output constraints to independently query the zero-shot VLM for each imaging factor.

Augmentation Category & Type	Lens	Sensor	Domain	Viewpoint
Target Classes	normal, fisheye	rgb, thermal, rgb-thermal, event, gated	real, simulation, video-game	front, back, side, pole, drone
Geometric
Horizontal flip	✓	✓	✓	Mild
Vertical flip	✓	✓	✓	—
Random rotation & translation	Mild	Moderate	Moderate	Mild
Color / Appearance

Table 2: Data augmentations for I-FAA classifier training. Category-specific, non-overlapping image augmentations applied during I-FAA (ours) training.

Panel of Table 3 — Table 3: Qualitative comparison of novel combinations. Image generation with unseen factor combinations. Our method, X-MULTI shows better factor adherence.

Panel of Table 4 — Table 4: Qualitative comparison of novel factor combinations with ControlNets. Image-to-image generation using ControlNet guidance with altered factors, showing X-MULTI (ours) achieves best factor adherence.

Method	Image Generation Metrics						Improved-Factor Alignment Accuracy (I-FAA)
	IS [ 34 ] $\uparrow$	CLIP Score [ 16 ] $\uparrow$				DS [ 46 ] $\uparrow$	Lens $\uparrow$	Sensor $\uparrow$	Domain $\uparrow$	View $\uparrow$	Avg. $\uparrow$
	IS [ 34 ] $\uparrow$	factor	context	average	full-prompt	DS [ 46 ] $\uparrow$	Lens $\uparrow$	Sensor $\uparrow$	Domain $\uparrow$	View $\uparrow$	Avg. $\uparrow$
DreamBooth [ 32 ]	3.68	23.96	23.35	23.66	24.67	0.55	0.78	0.24	0.48	0.27	0.44
SDXL Zeroshot [ 30 ]	3.11	22.31	25.59	23.95	27.80	0.59	0.72	0.25	0.29	0.29	0.39
MULTI [ 13 ]	3.28	24.20	23.43	23.82	27.32	0.65	0.63	0.29	0.64	0.34	0.47
X-MULTI (ours)	3.23	24.62	23.57	24.10	28.22	0.59	0.76	0.30	0.66	0.40	0.53

Table 5: Quantitative comparison on novel factor combinations. Performance metrics across methods with and without ControlNet structural guidance, with best values highlighted in bold. X-MULTI (ours) achieves best overall disentanglement.

Panel of Table 6 — Table 6: Comparison of attention maps for FAA and I-FAA. Grad-CAM visualizations showing that I-FAA (ours) learns more semantically meaningful attention regions.

Figure 2: Cramer’s V correlations. Pairwise correlations between predicted factors for FAA (left) and I-FAA (ours) (right), showing that I-FAA reduces cross-factor information leakage. High values show strong correlations.

Metric	Lens	Sensor	Domain	View	Avg.
FAA [ 13 ]	0.23	0.83	0.50	0.17	0.43
I-FAA	1.00	0.97	0.98	0.84	0.95

Table 7: FAA and I-FAA on ground-truth DF-RICO images. Comparison of FAA and I-FAA (ours) metric on ground-truth images.

Panel of Table 8 — Table 8: Factor prediction comparison. Per-factor classification outputs compared against ground-truth labels for ground truth images. I-FAA (ours) predicts factor correctly.

Panel of Figure 3 — Figure 3: Confusion matrices of the zero-shot VLM for factor classification: (a) Lens, (b) Sensor, (c) Domain, and (d) Viewpoint. Rows correspond to ground-truth labels and columns to predicted labels.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 12 — Table S.1 : Qualitative comparison of existing combinations. X-MULTI (ours) faithfully reproduces all target sensor, lens, and domain modalities present in the training distribution.

Method	Image Generation Metrics							Improved-Factor Alignment Accuracy (I-FAA)
	FID [ 17 ] $\downarrow$	IS [ 34 ] $\uparrow$	CLIP Score [ 16 ] $\uparrow$				DS [ 46 ] $\uparrow$	Lens $\uparrow$	Sensor $\uparrow$	Domain $\uparrow$	View $\uparrow$	Avg. $\uparrow$
	FID [ 17 ] $\downarrow$	IS [ 34 ] $\uparrow$	factor	context	average	full-prompt	DS [ 46 ] $\uparrow$	Lens $\uparrow$	Sensor $\uparrow$	Domain $\uparrow$	View $\uparrow$	Avg. $\uparrow$
DreamBooth [ 32 ]	58.27	5.02	24.52	23.75	24.64	24.86	0.57	1.00	0.91	0.96	0.80	0.91
SDXL Zeroshot [ 30 ]	60.19	4.36	20.70	24.68	21.19	25.13	0.59	0.65	0.71	0.65	0.36	0.59
Inspiration Tree [ 39 ]	55.35	5.08	19.78	21.89	20.84	23.78	0.59	0.60	0.64	0.68	0.42	0.58
MULTI [ 13 ]	60.13	4.68	23.82	24.77	24.29	27.30	0.56	0.98	0.90	0.94	0.82	0.91

Appendix

Table S.2 : Quantitative comparison on existing factor combinations. Best in bold. X-MULTI (ours) achieves the highest I-FAA metric while maintaining highly competitive underlying image generation quality scores.

Factor	Augmentation Operator	Exact Transformation
Lens	Random Rotation	Max $\pm 5^{\circ}$ (Probability $p=0.5$ )
	Random Affine Translation	Max $\pm 3\%$ horizontally/vertically ( $p=0.5$ )
	Color Jittering	Brightness/Contrast/Saturation: $\pm 0.2$ , Hue: $\pm 0.1$ ( $p=0.5$ )
	Grayscale Conversion	Probability $p=0.1$
	Gaussian Blur	Kernel Size: $5\times 5$ , Sigma Range: $[0.1,2.0]$ ( $p=0.5$ )
	Noise & Occlusions	Gaussian Noise (Std: $0.05$ ), Random Erasing ( $p=0.1$ ), Cutout ( $p=0.1$ ), HideAndSeek ( $6\times 6$ grid, $p=0.1$ )

Appendix

Table S.3 : Detailed operational parameter ranges for factor classifiers of I-FAA (ours).

Figure S.1 : Cross-factor accuracy variance analysis. Individual factor prediction accuracies evaluated under ground-truth class conditioning. I-FAA (ours) has less cross-factor variance.

Figure S.2 : Error co-occurrence patterns. Conditional joint error probabilities map the independence of classification failures, showing I-FAA (ours) has low co-occurrence.

Panel of Table 17 — Table S.4 : Extended qualitative comparison of attention maps between FAA and I-FAA.

Disentangled Representation Learning

Disentangled representation learning separates distinct factors of variation so models can interpret, control, and generalize their predictions.

See all 232 papers

MULTI: Disentangling Camera Lens, Sensor, View, and Domain for Novel Image Generation
May 12, 2026Sonali Godavarthy, Matthias Neuwirth-Trapp, Tim-Felix Faasch +3Disentangled Representation Learning Compositional Image Generation
Bosch Research, Hildesheim, Germany · ETH Zürich, Zürich, Switzerland · University of Siegen, Siegen, Germany +1
Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Models
Aug 19, 2025Yeji Park, Minyoung Lee, Sanghyuk Chun +1Large Vision-Language Models VLM Reasoning
Sogang University · Princeton University · NAVER AI Lab
Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks
Mar 4, 2026Chenjun LiVLM Reasoning Multi-Image Reasoning
School of Electrical and Computer Engineering, Cornell University

May 12, 2026J/K move · Enter open · S save

MULTI: Disentangling Camera Lens, Sensor, View, and Domain for Novel Image Generation

Sonali Godavarthy, Matthias Neuwirth-Trapp, Tim-Felix Faasch, Maarten Bieshaar+2

Bosch Research, Hildesheim, Germany · ETH Zürich, Zürich, Switzerland · University of Siegen, Siegen, Germany+1

Disentangled Representation Learning Compositional Image Generation Controllable Image Generation

Open paper AskPDF arXiv

Recent text-to-image models produce high-quality images, yet text ambiguity hinders precise control when specific styles or objects are required. There have been a number of recent works dealing with learning and composing multiple objects and patterns. However, current work focuses almost entirely on image content, overlooking imaging factors such as camera lens, sensor types, imaging viewpoints, and scenes' domain characteristics. We introduce this new challenge as Imaging Factor Disentanglement and show limitations of current approaches in the regime. We, therefore, propose the new method Multi-factor disentanglement through Textual Inversion (MULTI). It consists of two stages: in the first stage, we learn general factors, and in the second stage, we extract dataset-specific ones. This setup enables the extension of existing datasets and novel factor combinations, thereby reducing distribution gaps. It further supports modifications of specific factors and image-to-image generation via ControlNets. The evaluation on our new DF-RICO benchmark demonstrates the effectiveness of MULTI and highlights the importance of Factor Disentanglement as a new direction of research.

Abstract

Disentangled Representation Learning

Explore similar work

MULTI: Disentangling Camera Lens, Sensor, View, and Domain for Novel Image Generation

Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Models

Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks