Generative Model via Quantile Assignment

Georgi Hrusanov; Oliver Y. Chén; Julien S. Bodelet

cs.LGFeb 20, 2026

Generative Model via Quantile Assignment

Authors: Georgi Hrusanov, Oliver Y. Chén, Julien S. Bodelet

Organizations: University of Lausanne, Lausanne, Switzerland · Lausanne University Hospital (CHUV), Lausanne, Switzerland · Department of Statistics, Stanford University, Stanford, USA

Abstract

Deep Generative models (DGMs) play two key roles in modern machine learning: (i) producing new information (e.g., image synthesis) and (ii) reducing dimensionality. However, traditional architectures often rely on auxiliary networks such as encoders in Variational Autoencoders (VAEs) or discriminators in Generative Adversarial Networks (GANs), which introduce training instability, computational overhead, and risks like mode collapse. We present NeuroSQL, a new generative paradigm that eliminates the need for auxiliary networks by learning low-dimensional latent representations implicitly. NeuroSQL leverages an asymptotic approximation that expresses the latent variables as the solution to an optimal transportation problem. Specifically, NeuroSQL learns the latent variables by solving a linear assignment problem and then passes the latent information to a standalone generator. We benchmark its performance against GANs, VAEs, and a budget-matched diffusion baseline on four datasets: handwritten digits (MNIST), faces (CelebA), animal faces (AFHQ), and brain images (OASIS). Compared to VAEs, GANs, and diffusion models: (1) in terms of image quality, NeuroSQL achieves overall lower mean pixel distance between synthetic and authentic images and stronger perceptual/structural fidelity; (2) computationally, NeuroSQL requires the least training time; and (3) practically, NeuroSQL provides an effective solution for generating synthetic data with limited training samples. By embracing quantile assignment rather than an encoder, NeuroSQL provides a fast, stable, and robust way to generate synthetic data with minimal information loss.

Figures & tables

Figure 1: A schematic representation of the neuroSQL architecture. Left: Algorithm for optimizing latent embeddings and generator parameters $\theta$ . Right: Data flow in neuroSQL for data synthesis. Here, “cost” refers to cost matrix entries: $C_{i,k}=\ell\big(X_{i},G_{\theta}(Q_{k})\big)$ solved by the linear assignment problem. “Momentum update” indicates: $\hat{Z}_{i}^{(t)}\leftarrow\rho\,Q_{\pi^{(t)}(i)}+(1-\rho)\,\hat{Z}_{i}^{(t-1)}$ .

Panel of Figure 2 — Figure 2: A Comparison of Latent Space of neuroSQL and VAE. Visualization of the two-dimensional latent space obtained from MNIST using neuroSQL (left) and VAE (right). NeuroSQL’s latent space forms more distinct, better-separated clusters than those of the VAE, whose clusters show more overlaps.

Figure 3: 2D brain images generated from neuroSQL (top row), VAE (middle row) and GAN (bottom row) with U-Net generator.

Dataset (Res.)	Model	pFID $\downarrow$	SSIM $\uparrow$	P
MNIST (28 $\times$ 28)	neuroSQL	0.61 $\pm$ 0.12	0.616 $\pm$ 0.074	2.79
	VAE	1.26 $\pm$ 0.26	0.179 $\pm$ 0.030	4.17
	GAN	2.00 $\pm$ 0.22	0.233 $\pm$ 0.018	5.56
	Diffusion	0.65 $\pm$ 0.34	0.492 $\pm$ 0.055	147.91
CelebA (64 $\times$ 64)	neuroSQL	5.81 $\pm$ 4.05	0.262 $\pm$ 0.036	7.54
	VAE	10.75 $\pm$ 7.92	0.196 $\pm$ 0.047	14.03

Table 1: Performance comparison across datasets (mean

\pm

std). Lower is better for proxy FID; higher is better for SSIM.

Method	Prec.	Rec.	Acc.
NeuroSQL	0.8788	0.8763	0.8771
VAE	0.7236	0.6705	0.6749

Table 2: Downstream classification performance on MNIST. A classifier was trained solely on synthetic data (10k samples) generated by neuroSQL and VAE and evaluated on real test data.

Panel of Figure 4 — Figure 4: Qualitative comparison of 36 randomly generated images for models trained on MNIST.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Computational scalability: mean epoch time versus sample size $N$ for VAE, GAN, and neuroSQL with different local assignment sizes $m$ .

Latent Dim	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
8	Greedy	10.764394	0.376178	0.296806
	Hungarian	11.121147	0.378670	0.274640
16	Greedy	11.632382	0.398841	0.241317
	Hungarian	11.798410	0.390295	0.240504
24	Greedy	13.222514	0.384961	0.272349
	Hungarian	11.338643	0.379329	0.271508

Appendix

Table 3: Ablation Study: U-Net Latent Dimensions

Latent Dim	Assign. Method	Method	Peak VRAM (MB)	RAM Used (MB)	Mean Epoch Time (s)	Mean Epoch GPU (MB)	Mean Epoch $\Delta$ RAM (MB)	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	greedy	NeuroSQL	81.5	2764	1.40	116	11.2	0.628	0.045	0.580
	hungarian	NeuroSQL	284.6	3078	1.37	293	0.39	0.576	0.041	0.616
		VAE	319.8	3091	1.33	320	0.10	1.259	0.050	0.249
		GAN	405.6	3115	1.56	406	0.17	2.060	0.064	0.223
3	greedy	NeuroSQL	284.8	3206	1.39	293	0.14	0.567	0.040	0.574
	hungarian	NeuroSQL	284.9	3298	1.38	293	0.00	0.556	0.037	0.607

Appendix

Table 4: Ablation: Assignment Methods and Resource Usage on MNIST (seed=11). Metrics averaged over epochs where applicable.

Latent dimension	Method	FID (Proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	7.914	0.390	0.259
	VAE	8.198	0.410	0.246
	GAN	9.936	0.450	0.218
4	NeuroSQL	9.241	0.403	0.257
	VAE	8.720	0.463	0.212
	GAN	19.420	0.509	0.206

Appendix

Table 5: Results on OASIS, a brain imaging dataset, using ConvNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM
16	NeuroSQL	25.410	0.249	0.301
	VAE	51.139	0.309	0.171
	GAN	168.993	0.727	0.008
32	NeuroSQL	31.957	0.257	0.242
	VAE	66.045	0.362	0.128
	GAN	158.103	0.687	0.008

Appendix

Table 6: Results on OASIS, a brain imaging dataset, using ResNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
4	NeuroSQL	8.519902	0.386861	0.254752
	VAE	7.858056	0.377805	0.265082
	GAN	26.234322	0.597633	0.206676
8	NeuroSQL	8.579974	0.401890	0.238080
	VAE	5.976874	0.384821	0.264277
	GAN	35.444614	0.609803	0.172225

Appendix

Table 7: Results on OASIS, a brain imaging dataset, using a U-Net across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
NeuroSQL	8.559346	0.387453	0.258766
VAE	6.772966	0.396473	0.254754
GAN	28.591262	0.605914	0.193429

Appendix

Table 8: Results on OASIS, a brain imaging dataset, using a U-Net — with results averaged across latent dimensions

\{4,8,16,32,64,128\}

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	9.64453	0.22094	0.31376
	VAE	11.49087	0.24068	0.27396
	GAN	26.56548	0.47277	0.10216
4	NeuroSQL	6.99789	0.22845	0.31645
	VAE	8.57467	0.23091	0.26471
	GAN	29.68144	0.49874	0.06955

Appendix

Table 9: Results on CelebA, a face attributes dataset, using ConvNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	4.40599	0.18367	0.27506
	VAE	8.01552	0.19720	0.24972
	GAN	19.01699	0.20240	0.13535
4	NeuroSQL	4.47511	0.16968	0.26754
	VAE	6.03891	0.20652	0.20760
	GAN	14.21587	0.26898	0.14515

Appendix

Table 10: Results on CelebA, a face attributes dataset, using ResNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	4.96169	0.18253	0.27537
	VAE	7.33227	0.19269	0.25696
	GAN	14.09393	0.22041	0.12822
4	NeuroSQL	3.54758	0.18932	0.27993
	VAE	4.99991	0.19061	0.22841
	GAN	7.66398	0.19251	0.14899

Appendix

Table 11: Results on CelebA, a face attributes dataset, using U-Net (unconditional) across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	22.023664	0.563209	0.339032
	VAE	44.276279	0.531442	0.284219
	GAN	74.178604	0.693773	0.080698
4	NeuroSQL	35.913769	0.538813	0.357857
	VAE	35.430927	0.527985	0.285893
	GAN	48.706165	0.669665	0.129869

Appendix

Table 12: Results on AFHQ, an animal faces dataset, using ConvNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
NeuroSQL	32.19	0.544	0.354
VAE	71.13	0.560	0.233
GAN	58.36	0.720	0.093

Appendix

Table 13: Results on AFHQ, an animal faces dataset, using ConvNet — with results averaged across latent dimensions.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	7.267559	0.459967	0.267403
	VAE	15.259568	0.448991	0.235907
	GAN	39.100639	0.550003	0.091928
4	NeuroSQL	11.560558	0.460007	0.246023
	VAE	13.387914	0.432443	0.192707
	GAN	37.873501	0.356168	0.120893

Appendix

Table 14: Results on AFHQ, an animal faces dataset, using ResNet across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
NeuroSQL	12.40	0.460	0.247
VAE	12.27	0.394	0.202
GAN	29.35	0.435	0.104

Appendix

Table 15: Results on AFHQ, an animal faces dataset, using ResNet — with results averaged across latent dimensions.

Latent dimension	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	NeuroSQL	17.626825	0.595854	0.272737
	VAE	10.595321	0.620105	0.279837
	GAN	23.709433	0.405830	0.116950
3	NeuroSQL	10.193194	0.617973	0.278050
	VAE	29.156761	0.578171	0.173115
	GAN	17.098032	0.607903	0.074634

Appendix

Table 16: Results on AFHQ, an animal faces dataset, using U-Net across latent dimensions. Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
NeuroSQL	12.486547	0.618680	0.268553
VAE	34.046180	0.578176	0.134123
GAN	50.295575	0.609927	0.047645

Appendix

Table 17: Results on AFHQ, an animal faces dataset, using U-Net — with results averaged across latent dimensions

\{2,3,4,8,16,32,64\}

Latent dimension	Seed	Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
2	11	NeuroSQL	0.696835	0.039503	0.564344
		VAE	1.070473	0.058065	0.200167
		GAN	2.157639	0.054278	0.245780
3	11	NeuroSQL	0.527451	0.030257	0.668574
		VAE	1.443453	0.060009	0.157831
		GAN	1.849820	0.057282	0.220785

Appendix

Table 18: Ablation results on MNIST, a database of handwritten digits (all runs). Lower is better for FID (proxy) and LPIPS; higher is better for SSIM.

Method	FID (proxy) $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
NeuroSQL	0.612143	0.034880	0.616459
VAE	1.256963	0.059037	0.178999
GAN	2.003730	0.055780	0.233283

Appendix

Table 19: Results on MNIST, a database of handwritten digits — averaged over latent dimensions

\{2,3\}

(seed

=11

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 493 papers

GEAR: Guided End-to-End AutoRegression for Image Synthesis
Jun 30, 2026Bin Lin, Zheyuan Liu, Chenguo Lin +8Vector Quantization Autoregressive Image Generation
Peking University · Tencent Hunyuan
AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions
May 13, 2026Ishika Agarwal, Sofia Stoica, Emre Can Acikgoz +4Synthetic Data Generation Synthetic Data Generation for Language Models
University of Illinois Urbana-Champaign1 · Amazon2
Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors
Oct 14, 2026Junhoo Lee, Hyeonjin Kim, Sangbum Han +1Generative Modeling Image Generation
Seoul National University

Jun 30, 2026J/K move · Enter open · S save

GEAR: Guided End-to-End AutoRegression for Image Synthesis

Bin Lin, Zheyuan Liu, Chenguo Lin, Sixiang Chen+7

Peking University · Tencent Hunyuan

Vector Quantization Autoregressive Image Generation Representation Alignment

Open paper AskPDF arXiv

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.

Abstract

Generative Modeling

Explore similar work

GEAR: Guided End-to-End AutoRegression for Image Synthesis

AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions

Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors