Image Generation Models: A Technical History

Rouzbeh Shirvani

Fig. 1: Comparison of AE latent density and VAE sampled latent

z

. VAE has a smooth and favorable latent space compared to the latent space in AE. Image source: [ 5 ]

Fig. 2: Diagram of the end to end VAE model architecture with reparameterization trick

Fig. 3: PixelCNN and conditioning on prior pixels. Image adapted from [ 10 ]

Fig. 4: PixelVAE utilizing PixelCNN to generate sharper images. Image adapted from [ 11 ]

Fig. 5: Conditional VAE: One can think of

X

as the condition upon which the output is being generated.

Box 1 Simplified Training and Inference Procedure for the Conditional VAE

Fig. 6: Left side: Conventional Variational Autoencoders. Right side: DRAW Network. Note the iterative process through which the model improves the output of the previous time step. Image adapted from [ 15 ]

Fig. 7: Iterative generation process utilized in DRAW networks. Note how the pixels become sharper through time. Red squares signify the area attended by the network at each time step. Image adapted from [ 15 ]

Fig. 8: Simplified diagram of the end to end VQ-VAE model architecture. Image adapted from [ 16 ]

Box 2 Simplified Training Steps for the VQ-VAE

Fig. 9: On the right, hierarchical models induce dependence among the latent variables. High-level latents (coarse) set the context for the low-level (fine-detail) latents. Image adapted from [ 17 ]

Fig. 10: Sample random images generated by NVAE. Image adapted from [ 21 ]

Fig. 11: Diagram of VDVAE architecture with top-down VAE architecture with residual blocks. Image adapted from [ 23 ] .

Fig. 12: Training pipeline for Generative Adversarial Networks

Box 3 Simplified Training Steps for the original GAN paper

Fig. 13: Sample Images Drawn from the Generator

G

. Image source: [ 24 ]

Fig. 14: Sample Images Drawn from the Generator

G

of DCGANs on LSUN dataset [ 26 ]

Fig. 15: Results confirming that GAN’s latent space is organized linearly

Fig. 16: WGAN provides a clear and crisp gradient everywhere. Figure adapted from [ 30 ]

Box 4 Training Algorithm for Wasserstein GAN with Gradient Penalty (WGAN-GP) [ 31 ]

Fig. 17: Overall Diagram for Conditional GANs (CGANs) and Auxiliary Classifier GANs (AC-GANs)

Fig. 18: MNIST digits generated are conditioned on labels

0,1,\ldots,9

respectively. Figure adapted from [ 35 ]

Method	Inception Score
AC-GANs	28.5 $\pm$ 0.20
concat	21.1 $\pm$ 0.35
projection	29.7 $\pm$ 0.61
projection (850K iterations)	36.8 $\pm$ 0.44

TABLE I: Comparing the Inception Scores of different conditional GAN models on the ImageNet

Fig. 19: Sample ”Typewriter” images generated using the concat and projection models. Note the higher quality of images in the projection model and how all the images in the concat model look the same. Figure adapted from [ 37 ] .

Fig. 20: (Top) Karras et al. [ 38 ] start from low-resolution images and progressively increase the complexity of the models and the size of the input images. (Bottom) Sample images generated from this approach. Image adapted from [ 38 ] .

Box 5 Style Mixing in StyleGAN1

Fig. 21: (Top) StyleGAN1 architecture where

z

is mapped to an intermediate latent

w

. Early layers determine coarse structure, while later layers refine color and details. (Bottom) Sample pictures generated by StyleGAN. Image source: [ 39 ]

Fig. 22: Style mixing in StyleGAN1 demonstrates high-quality image generation by separating coarse and fine styles from two different image styles. Image source: [ 39 ]

Fig. 23: StyleGAN2 improves image quality and reduces artifacts compared to StyleGAN1. Image source: [ 40 ]

Fig. 24: In this image, the teeth should follow the pose, but they stay fixed at a specific pixel coordinate. Image adapted from StyleGAN2 [ 40 ] .

Dataset	Config	FID $\downarrow$	EQ-T $\uparrow$	EQ-R $\uparrow$
FFHQ-U	StyleGAN2	$3.79$	$15.89$	$10.79$
70,000 img, $1024^{2}$	StyleGAN3-T (ours)	$3.67$	$61.69$	$13.95$
Train from scratch	StyleGAN3-R (ours)	3.66	64.78	47.64
FFHQ	StyleGAN2	2.70	$13.58$	$10.22$
70,000 img, $1024^{2}$	StyleGAN3-T (ours)	$2.79$	$61.21$	$13.82$
Trained from scratch	StyleGAN3-R (ours)	$3.07$	64.76	46.62

TABLE II: StyleGAN3 improving FID and EQ scores compared to StyleGAN2. Results adapted from [ 41 ] .

Fig. 25: Example results by StackGAN, GAWWN [ 45 ] , and GAN-INT-CLS [ 46 ] .

Fig. 26: Results from AttnGAN. Images generated in 3 stages of generation. Note the most attended words in the 2nd and 3rd rows.

Fig. 27: Image on the left is a 4x upsampled version of the image on the right. Image source: [ 48 ]

Fig. 28: Overall diagram of a Normalizing Flow.

f

and

f^{-1}

share the same neural network parameters.

Box 6 Training and Generation Procedure for Normalizing Flows

Fig. 29: Unbiased samples from NICE(up) and Real NVP(down) trained on CIFAR-10. You can notice the improvement in image quality going from NICE to Real NVP.

Fig. 30: IAF VAE results in a clearer fit of posterior distribution compared to normal VAE.

Fig. 31: Glow architecture compared to RealNVP.

Fig. 32: Generated images of celebrities with Generative Flow (Glow).

Model	CIFAR-10	ImageNet 32 $\times$ 32	ImageNet 64 $\times$ 64
Real NVP	3.49	4.28	–
Glow	3.35	4.09	3.81
IAF-VAE	3.11	–	–
Flow++	3.08	3.86	3.69

TABLE III: Unconditional image modeling results in bits/dim. Table source [ 60 ] .

Fig. 33: Comparison between Glow and FFJORD in estimating distribution. Image source: [ 62 ] .

Fig. 34: Deterministic mapping in prior work versus SRFlow that learns the distribution of photorealistic images. Image source: [ 63 ] .

Fig. 35: TARFLOW (top) and STARFLOW(bottom) improve generation capabilities of Normalizing Flows. Image source: [ 64 , 65 ] .

PixelCNN	Row LSTM	Diagonal BiLSTM
$7\times 7$ conv mask A
Multiple residual blocks:
Conv $3\times 3$ mask B	Row LSTM i-s: $3\times 1$ mask B s-s: $3\times 1$ no mask	Diagonal BiLSTM i-s: $1\times 1$ mask B s-s: $1\times 2$ no mask
ReLU followed by $1\times 1$ conv, mask B (2 layers)
256-way Softmax for each RGB color (Natural images) or Sigmoid (MNIST)

TABLE IV: Architecture summary of PixelCNN and LSTM based PixelRNN. Table source: [ 67 ] .

Fig. 36: Sample images generated by PixelRNN. Image source: [ 67 ] .

Fig. 37: Sample images generated by Gated PixelCNN conditioned on class labels Lhasa Apso dog (top) and Brown bear (bottom). Image source: [ 9 ] .

Fig. 38: PixelSnail Model Architecture combining causal CNN with self attention. Architecture source: [ 69 ] .

Box 7 Training and Generation Procedure for an Autoregressive Image Transformer

Fig. 39: Query block size of

h_{q}\times w_{q}

. Memory block extends the query block upward by

h_{m}

and left/right by

w_{m}

pixels.

h_{m}=w_{m}=2

and

h_{q}=w_{q}=4

. Note how the current pixel,

q

, is attending the prior pixels. The future (masked) pixels are shown in white color. Image source: [ 70 ] .

Model	NLL
PixelCNN	3.14
Row PixelRNN	3.00
Gated PixelCNN	3.03
PixelCNN++	2.92
PixelSNAIL	2.85
Image Transformer	2.9

TABLE V: Negative log-likelihood (NLL) on CIFAR-10 (test).

Fig. 40: The simplified pre-training pipeline for iGPT. Image source: [ 71 ] .

Fig. 41: Zero-shot samples from DALL-E 1 on the CUB dataset. Image source: [ 72 ] .

Fig. 42: The training architecture for the Taming Transformers for High-Resolution Image Synthesis. Note how each part of the model is frozen during different training phases. Diagram source: [ 74 ] .

Fig. 43: Image completion for unconditional, depth to image, guided synthesis, and pose-guided. Image source: [ 74 ] .

Fig. 44: Sample images generated by the CogView style learning model. Image source: [ 76 ] .

Fig. 45: MaskGIT training pipeline. Image source: [ 77 ] .

Fig. 46: Given the images on the left and target class, MaskGIT replaces the bounding box region with the target class. Image source: [ 77 ] .

Fig. 47: Training pipeline in Muse. They added a second VQ tokenizer for increasing the resolution of the image. They brought in a pretrained language model (T5) for encoding input text. Image source: [ 78 ] .

Fig. 48: Training pipeline for Parti. Left side shows the autoregressive transformer based encoder-decoder. Right side is the ViT-VQGAN for image tokenizer/detokenizer. The super-resolution section at the end of ViT-VQGAN is omitted for simplicity. Diagram source [ 80 ] .

Fig. 49: Improved sample quality in Parti as the number of parameters in the autoregressive encoder–decoder increases from 750M to 20B. The input prompt is “A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says Welcome Friends!”. Image source: [ 80 ] .

Fig. 50: Top: Systematically destroying the structure in the image by gradually adding noise to the image. Bottom: Learned reverse diffusion process to restore the original image. Image source: [ 83 , 84 ] .

Box 8 Training and Generation Procedure for a Denoising Diffusion Probabilistic Model (DDPM).

Fig. 51: This figure shows unconditional progressive generation on CIFAR-10 using a diffusion model. Image source: adapted from [ 83 ] .

Model	ImageNet	CIFAR
Improved DDPM [ 87 ]	3.53	2.94
DDPM (cont. flow) [ 88 ]	–	2.99
DDPM [ 83 ]	3.77	3.70
Sparse Transformers [ 89 ]	3.44	2.80
Routing Transformers [ 90 ]	3.43	–
Pixel SNAIL [ 69 ]	3.52	2.85

TABLE VI: Negative log-likelihood (bits per dimension) of improved DDPM compared to other generative models on ImageNet and CIFAR.

Model	NFE	FID $\downarrow$	Precision $\uparrow$	Recall $\uparrow$
PD [ 91 ]	1	15.39	0.59	0.62
DFNO [ 94 ]	1	8.35	–	–
CD	1	6.20	0.62	0.63
PD [ 91 ]	2	8.95	0.63	0.65
CD	2	4.70	0.69	0.64

TABLE VII: ImageNet

64\times 64

results comparing progressive distillation (PD), Diffusion Fourier Neural Operator (DFNO), and consistency distillation (CD). NFE denotes the number of function evaluations.

Fig. 52: Progressive distillation by Salimans and Ho enables a diffusion model to generate final outputs in a considerably smaller number of steps. Image source: [ 91 ] .

Fig. 53: Consistency distillation learns to map any noisy point back to its origin

x_{0}

along the generative trajectory. Image source: [ 93 ] .

Fig. 54: Classifier-guided diffusion models beating BigGAN-deep in FID scores. Lower FID score indicates more realistic images, while a higher IS score indicates more diverse images. Image source: [ 95 ] .

Fig. 55: Top: samples from non-guided conditional sampling (

w=0

). Bottom: samples from classifier-free guidance with

w=3

. Image source: [ 97 ] .

Fig. 56: Overall training pipeline of latent space diffusion model where the diffusion happens on the latent space. Image source: [ 101 ] .

Fig. 57: Latent diffusion model (LDM) maps the input condition to the intermediate layers of UNet via cross attention and generates output images based on input condition. Here the input is conditioned on removing the highlighted parts of the image. Image source: [ 101 ] .

Fig. 58: Transformer GFLOPs vs. FID for DiT models. Larger transformers achieve lower (better) FID scores. Image source: [ 102 ] .

Fig. 59: Compute efficiency of DiT, i.e., larger models reach better FID scores with less training. Image source: [ 102 ] .

Fig. 60: Image uncropping results comparing Palette, Boundless, and InfinityGAN on the Places2 [ 106 ] dataset. Image source: [ 103 ] .

Fig. 61: The overall architecture for DALL-E 2. The authors gave it the name un-CLIP since they are effectively generating an image from an image embedding (the reverse direction of CLIP). Image source: [ 107 ] .

Box 9 Simplified training steps for the DALL-E 2 unCLIP model.

Fig. 62: A few sample

1024\times 1024

images from the un-CLIP (DALL-E 2) model. Image source: [ 107 ] .

Fig. 63: Oversimplified version of the Imagen model pipeline. Image source: [ 108 ] .

Model	Zero-shot FID-30K
DALL-E	17.89
GLIDE	12.24
DALL-E 2	10.39
Imagen	7.27

TABLE VIII: Zero-shot FID-30K scores on MS-COCO comparing Imagen to other text-to-image models.

Fig. 64: Image samples generated by Imagen (left) and DALL-E 2 (right). Image source: [ 108 ] .

Fig. 65: Top and bottom images show the original and upsampled image captions, respectively. Images generated by DALL-E 3. Image source: [ 109 ] .

Dataset	DALL-E 3	Midjourney 5.2	Stable Diffusion XL	DALL-E 2
DALL-E 3 Eval (prompt following)	153.3	-104.8	-189.5	–
DALL-E 3 Eval (style)	74.0	30.9	-95.7	–
MSCOCO (coherence)	71.0	48.9	-84.2	–
Drawbench	61.7	–	-34.0	-79.3

TABLE IX: Comparing DALL-E 3 human evaluation performance against different image generation models across several datasets. Table from [ 109 ]

Fig. 66: Two-stage pipeline for SDXL: a base model followed by a refinement model for high-resolution image synthesis. Image source: [ 110 ] .

Model	SDXL	SD 1.4/1.5	SD 2.0/2.1
# of UNet params	2.6B	860M	865M
Transformer blocks	[0, 2, 10]	[1, 1, 1, 1]	[1, 1, 1, 1]
Channel mult.	[1, 2, 4]	[1, 2, 4, 4]	[1, 2, 4, 4]
Text encoder	CLIP ViT-L & OpenCLIP ViT-bigG	CLIP ViT-L	OpenCLIP ViT-H
Context dim.	2048	768	1024
Pooled text emb.	OpenCLIP ViT-bigG	N/A	N/A

TABLE X: Key architectural differences between SDXL and earlier Stable Diffusion (SD) models.

Fig. 67: Sample images generated by SDXL versus Stable Diffusion (SD) v1.5 and v2.1. Image source: [ 110 ] .

Fig. 68: Similarity between residual networks and ODE-based networks. A residual network composes a sequence of discrete transformations of the hidden state, whereas an ODE network defines a continuous vector field whose flow transports the state over time. Image source: [ 111 ] .

Box 10 Forward Euler inference procedure for Rectified Flow: start from

Z_{0}\sim\pi_{0}

and integrate

\frac{dZ_{t}}{dt}=v_{\theta}(Z_{t},t)

to obtain

Z_{1}

.

Fig. 69: Rectified Flow and reflow trajectories. The top two rows illustrate noise to image transport under Rectified Flow. The second row shows reflow that produces straighter paths that can be traversed with fewer numerical integration steps. The bottom rows show an analogous effect for image to image transport, where reflow similarly reduces curvature and improves sampling efficiency. Image source: [ 112 ]

Fig. 70: Model architecture for Scaling Rectified Flows that are based on Multimodal Diffusion Transformers (MM-DiT). Image source: [ 113 ]

Fig. 71: Sample images generated by Scaling Rectified Flow Transformers, along with their corresponding prompts. Image source: [ 113 ]

Box 11 A simplified overview of Flow Matching training (via regression to a target velocity along a chosen conditional path) and inference (via an Euler sampler).

	CIFAR-10			ImageNet $32\times 32$
Model	BPD $\downarrow$	FID $\downarrow$	NFE $\downarrow$	BPD $\downarrow$	FID $\downarrow$	NFE $\downarrow$
Ablations
DDPM	3.12	7.48	274	3.54	6.99	262
Score Matching	3.16	19.94	242	3.56	5.68	178
ScoreFlow	3.09	20.78	428	3.55	14.14	195
Ours

TABLE XI: Flow Matching results compared to diffusion baselines. FM with Optimal Transport achieves better likelihood (BPD), sample quality (lower FID), and fewer function evaluations (lower NFE) than diffusion based DDPM.

Fig. 72: The overall pipeline for VideoGAN. The two foreground and background streams are combined to generate the output video. Image source: [ 119 ]

Fig. 73: Sample generated video frames from VGAN showing people walking on a golf course. Image source: [ 119 ]

Fig. 74: Temporal video generation (TGAN) model pipeline. Image source: [ 120 ]

Fig. 75: The overall model pipeline for MoCoGAN. Image source: [ 121 ]

User preference, %	Facial Exp.	Tai-Chi
MoCoGAN / VGAN	84.2 / 15.8	75.4 / 24.6
MoCoGAN / TGAN	54.7 / 45.3	68.0 / 32.0

TABLE XII: User preference comparison between the generated videos by MoCoGAN, VGAN, and TGAN [ 120 ] on Facial Expression [ 122 ] and Tai-Chi [ 123 ] datasets. Table source: [ 121 ] .

Fig. 76: Two stage training pipeline for VideoGPT. In stage 1, the VQ-VAE is trained and in stage 2 a GPT-style transformer is trained over latent codebooks. Image source: [ 124 ]

Method	Resolution	FID ( $\downarrow$ )	IS ( $\uparrow$ )
MoCoGAN [ 121 ]	16x64x64	26998 $\pm$ 33	12.42
TGAN-F [ 127 ]	16x64x64	8942.63 $\pm$ 3.72	13.62
TGAN-ODE [ 128 ]	16x64x64	26512 $\pm$ 27	15.2
TGAN-F [ 127 ]	16x128x128	7817 $\pm$ 10	22.91 $\pm$ .19
VideoGPT [ 124 ]	16x128x128	—	24.69 $\pm$ 0.30
TGAN-v2 [ 129 ]	16x64x64	3431 $\pm$ 19	26.60 $\pm$ 0.47

TABLE XIII: Comparing the inception score (IS) and Fréchet Inception Distance (FID) of different video generation methods on UCF-101 dataset. Table source: [ 126 ] .

Fig. 77: Text-conditioned video frames generated by the diffusion video model via a two stage diffusion model. Stage 1 generates the

16\times 64\times 64

lower resolution version and stage 2 uses a second model for simultaneous super-resolution and autoregressive extension to

64\times 128\times 128

. Image source: [ 126 ]

Box 12 Simplified training algorithm for Stable Video Diffusion.

Fig. 78: Text-to-video samples from the stable video diffusion (SVD) model. Captions from top to bottom are: “A hiker is reaching the summit of a mountain, taking in the breathtaking panoramic view of nature.”, “A unicorn in a magical grove, extremely detailed.”, “Shoveling snow”, “A beautiful fluffy domestic hen sitting on white eggs in a brown nest, eggs are under the hen.”, and “A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh”. Image source: [ 131 ]

Fig. 79: Simplified training pipeline for Make-A-Video. Input text gets mapped to image embedding and a desired fps by the prior

P

. It then goes through decoder

D_{t}

that generates

16\times 64\times 64

RGB frames. Toward the end, it goes through spatial and temporal resolution blocks. Image source: [ 132 ]

Fig. 80: Sample video frames generated by Make-A-Video model for different applications such as generation, animation, and interpolation. Image source: [ 132 ]

Fig. 81: Overall simplified training pipeline for Imagen Video. In practice, the text encoder is injected into all the models. SSR and TSR are spatial and temporal super-resolution diffusion models. Image source: [ 133 ]

Fig. 82: Sample video frames generated by Imagen Video. Note the model’s understanding of 3-dimensional structure. Image source: [ 133 ]

Fig. 83: Lumiere model pipeline. Unlike Imagen Video, the Lumiere pipeline processes all the incoming frames at once without a cascade of models which allows it to learn a globally coherent motion. Image source: [ 135 ]

Fig. 84: Lumiere applied to the first few frames of video from Imagen Video. Top row shows the X-T slice from the original Imagen Video and bottom row shows the X-T slice from the Lumiere. Note how Lumiere has a more coherent X-T slice. Image source: [ 135 ]

Fig. 85: AI generated images have become increasingly photorealistic over time. Image source: [ 139 ]

Fig. 86: Transferring the source to the target expression in facial reenactment. Image source: [ 146 ]

Fig. 87: Overall pipeline for facial reenactment. Image source: [ 146 ]

Fig. 88: They first convert audio into a sparse, time-varying mouth shape, then generate a photorealistic mouth texture and composite it onto the target video. They also match and re-time the mouth sequence to the video so head motion looks natural and fits the speech. Image source: [ 148 ]

Fig. 89: Overall pipeline for detecting blinking in fake face videos. (a) original sequence (b) sequence after alignment (c) LRCN: feature extraction, sequence learning, and state prediction. Image source: [ 149 ]

Methods	UADFV	DeepfakeTIMIT
		LQ	HQ
Two-stream NN [ 153 ]	85.1	83.5	73.5
Meso-4 [ 154 ]	84.3	87.8	68.4
MesoInception-4 [ 154 ]	82.1	80.4	62.7
HeadPose [ 151 ]	89.0	–	–
VGG16-backbone	84.5	84.6	57.4

TABLE XIV: DeepFake detection AUC performance comparison on UADFV [ 151 ] and DeepfakeTIMIT [ 152 ] datasets. Table source: [ 150 ]

Fig. 90: End-to-end pipeline for training a model for detecting GAN generated images. Image source: [ 155 ]

Training dataset	Testing dataset	Accuracy
CycleGAN	StarGAN	99.49
StarGAN	cycleGAN	93.42

TABLE XV: Cross-dataset detection accuracy when training on CycleGAN vs StarGAN.

Fig. 91: Comparison between the spectrum of a real image and a generated image. Image source: [ 164 ]

Fig. 92: Impact of up convolutional layers in the spectral response of the GAN/VAE generated images. Image source: [ 165 ]

Fig. 93: The overall pipeline for Diffusion Reconstruction Error (DIRE). The input image (

x_{0}

) goes through DDIM inversion, DDIM reconstruction, and DIRE process respectively. Image source: [ 166 ]

Fig. 94: Real images tend to have larger DIRE values compared to diffusion generated images. Image source: [ 166 ]

Fig. 95: Stable signature steps. (a) Pre-training the watermark encoder and decoder. (b) Finetuning the diffusion decoder. (c) Generating an image with invisible watermarks. Image source: [ 167 ]

Image Generation Models: A Technical History

Abstract

Figures & tables

Explore similar work

Evolution of Video Generative Foundations

Fully AI-Generated Image Detection: Definition, Recent Advances and Challenges

Video Generation Models: A Survey of Post-Training and Alignment