Improving Generative Model Self-Training with Geometrically Modified Outputs

Patrick Batsell; Thomas Walker; Richard Baraniuk

cs.LGSep 28, 2026

Improving Generative Model Self-Training with Geometrically Modified Outputs

Authors: Patrick Batsell, Thomas Walker, Richard Baraniuk

Organizations: Department of Electrical and Computer Engineering Rice University Houston, TX 77005, USA

Abstract

Self-training generative models - the continued improvement of a model using its own outputs - is becoming increasingly important as high-quality training data becomes scarce. However, naively finetuning on model-generated samples leads to degradation through model collapse and the model autophagy disorder. Negative-guidance self-training methods turn this degradation into a useful signal, using a model finetuned on its own outputs to guide the original model toward improved generation. Existing methods, however, take the negative signal in standard model outputs as given. We instead ask whether this signal can be explicitly strengthened. We introduce Geometrically Modified Outputs (GMOs), which reweight the singular values of the generator's input-output Jacobian to increase the influence of its leading singular directions. This geometric modification amplifies the mode-seeking behavior and distortions of standard outputs, providing a stronger and more targeted negative signal for self-training. Across a range of one-step generative models, GMOs consistently improve the performance of negative-guidance methods, including Neon and SIMS, compared with using standard model outputs.

Figures & tables

Panel of Figure 1 — Figure 1: Geometrically Modified Outputs (GMOs) improve the performance of one-step generative models through self-training. Here we present examples of GMOs for the IMM ( Zhou et al., 2025 ) one-step generative model trained on Imagenet256 ( Krizhevsky et al., 2012 ) . In the first panel, we show the model’s standard outputs; in the center panel, we show the perturbations of these outputs that yield the corresponding GMOs in the right panel (see Figure 11 for more examples). Using GMOs in self-training algorithms more effectively improves generative models.

Panel of Figure 2 — Figure 2: The geometry of a generative model collapses under naïve self-training. Here we finetune a MeanFlow SiT-B/2 model pre-trained on ImageNet256 for 50 epochs on $30,000$ of its own outputs. Throughout training, we monitor the FID ( Heusel et al., 2017 ) of the model (left), and its effective rank on a collection of 16 fixed latent vectors (right).

Panel of Figure 3 — Figure 3: Here we consider computing spectral statistics from the input-output Jacobians of a MeanFlow one-step generator ( Geng et al., 2025 ) with a UNet architecture ( Ronneberger et al., 2015 ; Song & Ermon, 2019 ) trained on CIFAR10 ( Krizhevsky & Hinton, 2009 ) . Ten latent vectors are sampled, and $E$ , $\sigma_{1}$ , ${\bm{u}}_{1}$ , and ${\bm{v}}_{1}$ are either computed exactly or using approximations. Power iteration is implemented $30$ times to obtain estimates for $\hat{\sigma}_{1}$ , $\hat{{\bm{u}}}_{1}$ , and $\hat{{\bm{v}}}_{1}$ . Similarly, $100$ samples are used to generate the Hutchinson estimate $\hat{E}$ . In the first panel, we record $\nicefrac{{\left|\hat{\sigma}_{1}-\sigma_{1}\right|}}{{\sigma_{1}}}$ . In the second panel, we record $1-\nicefrac{{\left\langle\hat{{\bm{u}}}_{1},{\bm{u}}_{1}\right\rangle}}{{\left\|\hat{{\bm{u}}}_{1}\right\|_{2}\left\|{\bm{u}}_{1}\right\|_{2}}}$ , and similarly for ${\bm{v}}_{1}$ . In the third panel, we record $\nicefrac{{\left|\hat{E}-E\right|}}{{E}}$ . In the fourth panel, we record the time taken to exactly compute the Jacobian and its singular value decomposition (Exact) and the time taken to perform $20$ power iterations and $20$ Hutchinson approximations (Approximate).

Panel of Figure 4 — Figure 4: GMOs provide meaningful perturbations to a model’s outputs that improve the effectiveness of Neon. Here, we consider applying the Neon self-training algorithm to the MeanFlow ( Geng et al., 2025 ) SiT-B/2 model ( Ma et al., 2024 ) trained on ImageNet256 ( Krizhevsky et al., 2012 ) . We generated collections of $30,000$ GMOs for alpha values $0.0$ (standard outputs), $0.05$ , and $0.2$ . We then finetuned the pre-trained checkpoint with a maximum compute budget of $1.8\times 10^{6}$ . At regular checkpoints, we apply Neon with different weight-merging parameters $w$ in the range $[0,2]$ . In the first and second panels, we show the minimum FID value achieved at each fine-tuning checkpoint with the corresponding weight-merging parameter $w$ , respectively. In the third panel, we visually compare standard outputs to GMOs and the Gaussian baseline (see Figure 15 for more examples).

Panel of Figure 5 — Figure 5: Neon with GMOs improves image quality and diversity. For the IMM 1-step generator ( Zhou et al., 2025 ) trained on ImageNet256 ( Krizhevsky et al., 2012 ) , we plot FID, precision, recall, density, and coverage ( Heusel et al., 2017 ; Kynkäänniemi et al., 2019 ) as a function of the GMO coefficient $\alpha$ . Here, $\alpha$ equal to zero corresponds to Neon ( Alemohammad et al., 2026 ) applied to standard outputs, while $\alpha>0$ corresponds to Neon applied to $\alpha$ -GMOs. Each curve reports the per- $\alpha$ optimum over fine-tuning budget $\mathcal{B}$ and merge weight $w$ . For more details, refer to Appendix D.2 .

Panel of Figure 6 — Figure 6: GMOs are transferable across generative models of different sizes and inference strategies. In the left panel, we consider using ImageNet256 GMOs from a SiT-B/2 MeanFlow model to improve a SiT-L/2 MeanFlow model using Neon. We compare this to Neon applied directly to the SiT-L/2 model’s outputs and GMOs. A computational fine tuning budget of $1.2\times 10^{6}$ is used in every case, and GMOs are generated with $\alpha$ equal to $0.1$ . In the right panel, we consider using CIFAR10 GMOs from a one-step IMM model to improve a 2-step IMM model using Neon. We compare this to Neon applied directly to the two-step IMM model’s outputs and GMOs. For more experimental detail on this particular experiment, refer to Appendix D.1 .

Architecture	Base FID	Neon w/ Standard Outputs			Neon w/ GMOs
		$\mathcal{B}$	$w$	FID	$\mathcal{B}$	$w$	$\alpha$	FID
IMM	$8.34$	$2.5\times 10^{6}(6.1\times 10^{-3}\%)$	$1.6$	$7.32(\pm 0.07)$	$2.87\times 10^{6}(7.0\times 10^{-3}\%)$	$1.6$	$0.2$	$\mathbf{6.25(\pm 0.03)}$
MeanFlow SiT-B/2	$6.08$	$7.2\times 10^{5}(0.25\%)$	$0.8$	$5.70(\pm 0.01)$	$4.8\times 10^{5}$ $(0.17\%)$	$1.2$	$0.05$	$\mathbf{5.60(\pm 0.01)}$
MeanFlow SiT-L/2	$3.97$	$1.8\times 10^{6}(0.63\%)$	$0.5$	$3.74(\pm 0.02)$	$1.8\times 10^{6}(0.63\%)$	$0.3$	$0.1$	$\mathbf{3.70(\pm 0.02)}$
AlphaFlow SiT-B/2	$5.55$	$4.8\times 10^{5}(0.16\%)$	$0.6$	$5.36(\pm 0.02)$	$4.8\times 10^{5}(0.16\%)$	$0.8$	$0.1$	$\mathbf{5.16(\pm 0.02)}$
AlphaFlow SiT-XL/2	$2.93$	$3\times 10^{5}(0.1\%)$	$1.2$	$2.64(\pm 0.02)$	$3\times 10^{5}(0.1\%)$	$1.1$	$0.05$	$\mathbf{2.59(\pm 0.02)}$

Table 1: The utilization of GMOs in the Neon self-training algorithm yields better generative models. Here, we implement the Neon self-training algorithm on various one-step generative models trained on ImageNet256. We consider Neon on the standard output of these generators, or on the corresponding GMOs, across various compute levels (as a percentage of pre-training compute), weight-merging parameters

w

, and

\alpha

values for GMOs (refer to Appendix D.2 for more details). Among the best-performing configurations, we report the average and standard deviation of the FID across five random seeds and

50,000

output samples. We provide a repository here containing the model checkpoints obtained using GMOs.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: GMOs amplify the “mode-seeking” nature of model outputs and their errors. Here we consider a collection of $2048$ GMOs for $\alpha$ values $0.0$ , $0.1$ , $0.2$ , $0.6$ , and $1.0$ generated using AlphaFlow ( Zhang et al., 2026 ) SiT-XL/2 ( Ma et al., 2024 ) model trained on ImageNet256 ( Krizhevsky et al., 2012 ) . In the top row, we visualize a t-SNE projection ( van der Maaten & Hinton, 2008 ) of the CLIP embeddings ( Radford et al., 2021 ) of the GMOs. In the bottom row, we visualize the GMOs for a specific latent vector.

Model	Standard Neon (% of pretraining)	Neon+GMOs (% of pretraining)
IMM	$0.007\%$	$0.01\%$
AlphaFlow-XL/2	$0.10\%$	$0.51\%$
AlphaFlow-B/2	$0.16\%$	$0.57\%$
MeanFlow-B/2	$0.26\%$	$0.61\%$
MeanFlow-L/2	$0.63\%$	$1.06\%$

Appendix

Table 2: Total pipeline cost as a percentage of pretraining compute, including the upfront cost of GMO generation.

Panel of Figure 8 — Figure 8: GMOs can reduce the finetuning required to achieve optimal results using Neon. Here we apply Neon to a UNet generative model ( Ronneberger et al., 2015 ) training on CIFAR10 ( Krizhevsky & Hinton, 2009 ) using the MeanFlow framework ( Geng et al., 2025 ) . We monitor the value of FID for a range of compute levels $\mathcal{B}$ and weight-merging parameters $w$ . With red markers, we indicate which combination of these hyperparameters yields the model with the lowest FID score.

Panel of Figure 9 — Figure 9: The complete set of evaluation curves for the experiment shown in the right panel of Figure 6 , and described in Appendix D.1 .

Method	FID $\downarrow$
Base IMM	$8.342\pm 0.058$
SIMS, standard outputs	$7.591\pm 0.057$
SIMS, GMOs ( $\alpha=0.1$ )	$\mathbf{6.920\pm 0.048}$

Appendix

Table 3: SIMS-style guidance on IMM ImageNet256. FID is reported as mean

\pm

sample standard deviation across five evaluations of fixed checkpoints, using

50{,}000

samples per evaluation. Both guided variants use

\omega=1.6

Architecture	Method	Precision	Recall	Density	Coverage
IMM (DiT-XL/2)	Base	$.585\pm.002$	$.647\pm.001$	$.652\pm.001$	$.662\pm.003$
	Neon-std	$.598\pm.002$	$.654\pm.002$	$.695\pm.001$	$.693\pm.002$
	GMOs	$.615\pm.003$	$.647\pm.002$	$\mathbf{.728\pm.003}$	$\mathbf{.714\pm.002}$
MeanFlow SiT-B/2	Base	$.717\pm.002$	$.452\pm.002$	$1.081\pm.006$	$.749\pm.001$
	Neon-std	$.720\pm.001$	$.452\pm.003$	$1.110\pm.006$	$.755\pm.001$
	GMOs	$.725\pm.001$	$.451\pm.003$	$\mathbf{1.124\pm.007}$	$\mathbf{.759\pm.002}$

Appendix

Table 4: Precision, recall, density, and coverage (5 seeds, mean

\pm

std). Density and coverage are maintained or improved from standard-outputs Neon to GMOs on every architecture.

Architecture	Base FID	Neon-std FID	GMOs FID	Welch $p$	GMOs’ reduction, % of Neon’s
IMM (DiT-XL/2)	$8.34$	$7.32\pm.07$	$\mathbf{6.25\pm.03}$	${\sim}10^{-9}$	$96\%$
MeanFlow SiT-B/2	$6.08$	$5.70\pm.01$	$\mathbf{5.60\pm.01}$	$2.6\times 10^{-7}$	$26\%$
MeanFlow SiT-L/2	$3.97$	$3.74\pm.02$	$\mathbf{3.70\pm.02}$	$0.013$	$17\%$
AlphaFlow SiT-B/2	$5.55$	$5.36\pm.02$	$\mathbf{5.16\pm.02}$	$2.6\times 10^{-7}$	$105\%$
AlphaFlow SiT-XL/2	$2.93$	$2.64\pm.02$	$\mathbf{2.59\pm.02}$	$0.004$	$17\%$

Appendix

Table 5: Statistical significance of the GMO improvement over standard-outputs Neon (5 seeds, mean

\pm

std). The final column reports the additional FID reduction of GMOs as a percentage of the reduction Neon itself achieves over the base model.

Panel of Figure 10 — Figure 10: Hyperparameter sweeps described in Appendix D.2 that yielded the results in Table 1 .

Panel of Figure 11 — Figure 11: Additional examples of those shown in Figure 1 at different GMO $\alpha$ levels.

Panel of Figure 12 — Figure 12: Qualitative comparison on IMM ImageNet-256 (Part I).

Panel of Figure 13 — Figure 13: In Figure 3 , we quantitatively demonstrate that power iteration can be used to accurately approximate the singular vectors of the input-output Jacobians of generative models. Here, we qualitatively support this by showing the exact (top) and approximate top left singular vectors of these input-output Jacobians.

Panel of Figure 14 — Figure 14: Here we provide additional examples to that illustrated in the bottom row of Figure 7 .

Panel of Figure 15 — Figure 15: Here we provide additional examples to complement the third panel of Figure 4 .

Explore similar work

May 9, 2026cs.CV

Improving Generative Adversarial Networks with Self-Distillation

In modern GANs, maintaining an Exponential Moving Average (EMA) of the generator's weights is a standard practice, as such an averaged model consistently outperforms the actively trained generator. However, the EMA generator is used for final deployment only and does not influence the training process. To address this missed opportunity, we introduce Self-Distilled GAN (SD-GAN) that employs the EMA generator as a teacher to guide the active generator (student) via perceptual loss. We prove the local asymptotic stability of SD-GAN in the Dirac-GAN setting and show that it dampens the parasitic cycling behavior that plagues the conventional GANs. Empirical evaluations across established architectures and datasets demonstrate that SD-GAN improves the final image quality on several metrics (FID and random-FID in particular), stabilizes the optimization trajectory and provides additional learning guidance that is not trivially correlated with the conventional adversarial loss. It also proves effective for fine-tuning pretrained GAN models.

Antoni Nowinowski, Krzysztof Krawiec

Poznan University of Technology

Conditional Gans Generative Adversarial Network Adversarial Training Exponential Moving Average

Ask PDF

Sep 30, 2026cs.AI

UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement

Modern multimodal models bring generation and understanding into a single unified system, which enables them to provide and learn from their own feedback. Motivated by this unified capacity, we introduce UniEvo-VL, a self-evolving framework for multimodal models to learn from this constructive self-correction feedback during test-time compute. Instead of relying on a separate, often larger, teacher, we leverage their self-critiques as privileged information and ask a single multimodal model to act as both teacher and student with different contexts. The student only sees the vanilla question, while the teacher conditions on the privileged critique. Then training minimizes the per-state divergence between their denoising diffusion distributions over the student's own sampling trajectories. Experiments demonstrate that UniEvo-VL improves the image generation capabilities of multimodal models, while maintaining their sensitivity to additional reflection information. Specifically, we build on top of the open-source Qwen-image-2512 and observe a significant performance gain from 0.747 to 0.808 on GenEval and from 32.97 to 35.53 on GenEval2 Soft-TIFA. Moreover, attempts with more powerful external critics (e.g., GPT5.6-Luna) show that multimodal models with strong judge capabilities can anticipate a higher self-evolving ceiling. Last but not least, mixed text-rendering outcomes show that our self-improvements may not be uniform across different tasks. Our study aims to shed light on the current hot recursive self-improvement research line to enhance the user experience when using multimodal models without external supervision or guidance.

Fang Wu, Da Xing, Yanjie Huang +16

Stanford University · Johns Hopkins University · Independent Researcher +7

Multimodal Model Self-Evolution Feedback Divergence

Ask PDF

Jul 29, 2026cs.LG

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

Alexi Gladstone, Heng Ji, Yilun Du

1UIUC · 2Harvard

Generative Models Pretraining Deep Learning Scaling

Ask PDF