cs.CVMar 31, 2026
SaveDiffusion Mental Averages
Organizations: VISTEC, Thailand
Abstract
Can a diffusion model produce its own "mental average" of a concept-one that is as sharp and realistic as a typical sample? We introduce Diffusion Mental Averages (DMA), a model-centric answer to this question. While prior methods aim to average image collections, they produce blurry results when applied to diffusion samples from the same prompt. These data-centric techniques operate outside the model, ignoring the generative process. In contrast, DMA averages within the diffusion model's semantic space, as discovered by recent studies. Since this space evolves across timesteps and lacks a direct decoder, we cast averaging as trajectory alignment: optimize multiple noise latents so their denoising trajectories progressively converge toward shared coarse-to-fine semantics, yielding a single sharp prototype. We extend our approach to multimodal concepts (e.g., dogs with many breeds) by clustering samples in semantically-rich spaces such as CLIP and applying Textual Inversion or LoRA to bridge CLIP clusters into diffusion space. This is, to our knowledge, the first approach that delivers consistent, realistic averages, even for abstract concepts, serving as a concrete visual summary and a lens into model biases and concept representation.
Figures & tables
Figure 2: Overview. Multiple noise latents are jointly optimized so that their denoising trajectories converge toward shared semantics. At each timestep , their -space activations are averaged to form a semantic target, and each latent is optimized to match it before denoising to the next step. Repeating this process across timesteps aligns coarse-to-fine semantics, yielding a single “mental average” of the concept.
Figure 3: Qualitative comparison across methods: GANgealing [ 70 ] , Avg. VAE , D 4 M [ 89 ] , MGD 3 [ 13 ] , and DMA (Ours) . Rows show concepts: astronaut , dog , bicycle , freedom . GANgealing cannot process abstract concepts, so its freedom cell is intentionally left blank.
| Consistency (↓) | Representativeness (↓) | ImageReward (↑) | |||||
| Method | CLIP | DreamSim | LPIPS | CLIP | DreamSim | LPIPS | |
| GANgealing † [ 70 ] | 0 | 0 | 0 | 0.386 | 0.477 | 0.851 | |
| Avg VAE | 0 | 0 | 0 | 0.473 | 0.806 | 0.805 | |
| D 4 M [ 89 ] | 0.168 | 0.274 | 0.572 | 0.197 | 0.363 | 0.672 | 0.823 |
| MGD 3 [ 13 ] | 0.180 | 0.319 | 0.643 | 0.195 | 0.364 | 0.687 | 0.755 |
| DMA (Ours) | 0.031 | 0.032 | 0.129 | 0.179 | 0.341 | 0.655 | 1.002 |
Table 1: Average scores across four categories ( Animal , Person , Object , Abstract ) on Consistency (↓), Representativeness (↓), and ImageReward (↑). GANgealing † is evaluated only on three non-abstract categories, as it lacks pretrained GANs on abstract concepts
Figure 4: Quality Trade-off. DMA achieves a superior balance of representativeness, consistency, and perceptual quality.
Figure 5: DMA prototypes of unsupervised modes. Top row: overall average. Bottom rows: averages of discovered modes.
Figure 6: DMA prototypes of grounded modes. We cluster astronaut by ethnicity, dog by breed, and car by model or color.
Figure 7: LoRA & Textual Inversion bridge the gap between the clustering and the -spaces, enabling cluster-specific prototypes.
Figure 8: DMA prototypes of various SD Variants. Soldier inherits model-specific gender bias, while Italy exhibits a persistent static bias toward a Venice-like scene across all variants.
Figure 9: DMA generalizes to the DiT architecture [ 69 ] .
Figure 10: Efficiency trade-off. We vary and , and show consistency scores (↓). Red is our default setting. Lower and trade consistency for speed and are up to the user.
Figure 11: Comparison with the pre-computed activation baseline. The pre-computed baseline (top) degrades over timesteps, while DMA (bottom) remains stable and consistent.
Figure 12: Comparison with the single-timestep optimization baseline. This baseline produces inconsistent results and introduces noticeable artifacts if optimization is done in later timesteps. In contrast, our method maintains both semantic consistency and visual quality.
Figure 13: Results of the replacement baseline with different . This baseline yields inconsistent images, even with high values.
Figure 14: Effects of different values on visual consistency. yields consistent images with reduced computational cost.
Figure 15: Quality trade-off across baseline hyperparameters. Across the full range of hyperparameter settings for each baseline, our method consistently achieves superior representativeness, consistency, and visual quality.
Figure 16: Effects of CFG scale on visual quality. Low CFG scales fail to generate faithful structures (as seen in monstera ), while excessively high CFG scales lead to oversaturated colors and unnatural contrast.
Figure 17: Effect of the number of noisy latents . In the blue box, we sample four distinct sets of noisy latents and compute their averages. For each set, we show eight example averages (in a 4x2 grid), each computed from a random latent in that set, which converge to similar-looking images within each set . However, at , noticeable variations are still observed across the four sets shown in the images with colored borders. As increases, averages across distinct sets become more similar, consistent with the statistical behavior of larger sample sizes.
Figure 18: Ablation on DiT transformer block selection for DMA. See details in Section C.7 .
Figure 19: Hyperparameter tuning of D 4 M for fair comparison. We sweep the noise-injection timestep across four concepts: bird , astronaut , car , and freedom . Larger values of preserve the averaged latent more strongly but lead to blurrier outputs, while smaller values make the result less consistent.
Figure 20: Hyperparameter tuning of MGD 3 for fair comparison. We sweep the mode-guidance weight across four concepts: bird , astronaut , car , and freedom . Larger increases attraction toward the estimated mode, improving structure but sometimes introducing over-sharpening or hallucinations.
| Method | Representativeness (↓) | Consistency (↓) | ImageReward (↑) |
|---|---|---|---|
| MGD 3 | 0.165 / 0.678 / 0.364 | 0.147 / 0.610 / 0.317 | -0.8151 |
| DMA (Ours) | 0.151 / 0.626 / 0.318 | 0.050 / 0.192 / 0.093 | -0.7085 |
Table 2: Quantitative comparison on DiT. (CLIP / LPIPS / DreamSim)
Figure 21: Average images of person , house , and cloth from different ethnicities.
Figure 22: Average images of a person with different attributes.
Figure 23: Average images of additional concepts with two different attributes.
Figure 24: Average images of various concepts.
Figure 25: Cluster averages for the Bicycle concept grounded by color . Top: Diffusion samples grouped into clusters according to the grounding attribute color . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 26: Cluster averages for the Cake concept grounded by type . Top: Diffusion samples grouped into clusters according to the grounding attribute type . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 27: Cluster averages for the Jellyfish concept grounded by color . Top: Diffusion samples grouped into clusters according to the grounding attribute color . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 28: Cluster averages for the Toy concept grounded by shape . Top: Diffusion samples grouped into clusters according to the grounding attribute shape . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 29: Cluster averages for the Nurse concept grounded by ethnicity . Top: Diffusion samples grouped into clusters according to the grounding attribute ethnicity . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 30: Cluster averages for the Place concept grounded by environment . Top: Diffusion samples grouped into clusters according to the grounding attribute environment . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 31: Cluster averages for the Object concept grounded by shape . Top: Diffusion samples grouped into clusters according to the grounding attribute shape . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 32: Cluster averages for the Building concept grounded by architecture . Top: Diffusion samples grouped into clusters according to the grounding attribute architecture . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 33: Cluster averages for the Person with musical instrument concept grounded by instrument . Top: Diffusion samples grouped into clusters according to the grounding attribute instrument . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
Figure 34: Cluster averages for the Person with pet concept grounded by pet . Top: Diffusion samples grouped into clusters according to the grounding attribute pet . Bottom: Cluster averages computed by DMA with LoRA. These averages highlight how our method captures the dominant visual characteristics within each cluster.
| Animal | Person | Object | Abstract | |||||||||
| Method | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ |
| GANgealing [ 70 ] | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | – | – | – |
| Avg VAE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| D 4 M [ 89 ] | 0.123 | 0.191 | 0.615 | 0.127 | 0.174 | 0.510 | 0.193 | 0.308 | 0.627 | 0.229 | 0.425 | 0.534 |
| MGD 3 [ 13 ] | 0.122 | 0.224 | 0.679 | 0.131 | 0.202 | 0.598 | 0.215 | 0.342 | 0.686 | 0.251 | 0.508 | 0.610 |
| DMA (Ours) | 0.015 | 0.021 | 0.138 | 0.022 | 0.019 | 0.118 | 0.040 | 0.039 | 0.158 | 0.047 | 0.049 | 0.101 |
Table 3: Consistency (↓) across concept categories. Lower values indicate higher consistency of prototypes within each seed. Each category includes representative concepts: Animal — Bird, Cat, Dog; Person — Astronaut, Doctor, Firefighter; Object — Bicycle, Car, TV Monitor; Abstract — Anger, Freedom, Poverty.
| Animal | Person | Object | Abstract | |||||||||
| Method | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ | CLIP↓ | DreamSim↓ | LPIPS↓ |
| GANgealing [ 70 ] | 0.310 | 0.394 | 0.839 | 0.523 | 0.599 | 0.866 | 0.323 | 0.439 | 0.848 | – | – | – |
| Avg VAE | 0.505 | 0.867 | 0.831 | 0.504 | 0.780 | 0.796 | 0.464 | 0.832 | 0.882 | 0.420 | 0.746 | 0.711 |
| D 4 M [ 89 ] | 0.158 | 0.320 | 0.708 | 0.144 | 0.230 | 0.613 | 0.224 | 0.379 | 0.714 | 0.260 | 0.522 | 0.652 |
| MGD 3 [ 13 ] | 0.155 | 0.310 | 0.718 | 0.140 | 0.230 | 0.636 | 0.223 | 0.375 | 0.724 | 0.262 | 0.543 | 0.671 |
| DMA (Ours) | 0.143 | 0.307 | 0.669 | 0.142 | 0.208 | 0.595 | 0.202 | 0.368 | 0.718 | 0.229 | 0.479 | 0.638 |
Table 4: Representativeness (↓) across concept categories. Lower values indicate closer alignment between the prototype and the overall concept distribution.
Figure 35: Baseline comparison for the astronaut concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 36: Baseline comparison for the firefighter concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 37: Baseline comparison for the doctor concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 38: Baseline comparison for the bird concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 39: Baseline comparison for the cat concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 40: Baseline comparison for the dog concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 41: Baseline comparison for the bicycle concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 42: Baseline comparison for the car concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
Figure 43: Baseline comparison for the tvmonitor concept. Top: Generated samples from the diffusion model. Bottom: Average images produced by GANgealing [ 70 ] , D 4 M [ 89 ] , MGD 3 [ 13 ] , and our method.
More in this topic
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,169 papers