TextSphere: Efficient Text-to-Image Generation via Shared Hyperspherical Alignment and Uniformity
Authors: Yixia Zhang, Girish A. Koushik, Xilu Wang, Wenwu Wang
Organizations: Department of Computer Science, University of Surrey, Guildford, United Kingdom · CVSSP, University of Surrey, Guildford, United Kingdom
Recent advances in text-to-image generation have substantially improved image quality, particularly through diffusion-based methods that require multiple inference steps. In contrast, Sphere Encoder offers an efficient generative framework capable of producing images with fewer forward steps. However, extending it to free-form text is nontrivial, because its class-conditioned design lacks cross-modal alignment. To address this challenge, we propose TextSphere, an efficient text-to-image generative framework that extends Sphere Encoder with language conditioning and cross-modal alignment. We incorporate pretrained CLIP text embeddings through adaptive layer normalization (AdaLN) to guide image generation while retaining the original spherical sampling mechanism. To bridge the cross-modal representation gap, we introduce a Spherical Semantic Alignment and Uniformity (SSAU) objective that jointly promotes semantic alignment and representation uniformity on the hypersphere. Experiments on CUB-200-2011, MS-COCO-2014, and Flickr30k demonstrate that TextSphere consistently reduces generation FID (gFID) over the text-conditioned baseline while achieving reconstruction FID (rFID) below 1.0. With only 4 and 6 sampling steps, TextSphere achieves lower gFID than the compared diffusion- and GAN-based models trained from scratch on CUB-200-2011 and MS-COCO-2014, respectively. We further observe that improved reconstruction fidelity does not necessarily yield better generation quality. In addition, we use Partial Least Squares Singular Value Decomposition (PLS-SVD) to analyze cross-modal alignment and modality gap, offering insight into the effectiveness of our approach.
Figures & tables
Figure 2: TextSphere Framework Overview. It consists of a frozen pretrained CLIP text encoder, an image encoder, and an image decoder. During training, the image encoder maps ground-truth images onto a spherical latent space via RMS normalization (Spherify), as defined in Eq. ( 1 ), while the CLIP text encoder extracts semantic embeddings t from the corresponding captions. The training objectives jointly optimize (1) contrastive alignment, (2) uniformity regularization, and (3) visual fidelity, with the decoder sampling random latent representations from the hypersphere to generate images.
Figure 3: Inference pipeline of TextSphere.
CUB-200-2011
MS-COCO-2014
Flickr30k
rFID ↓
rIS ↑
rFID ↓
rIS ↑
rFID ↓
rIS ↑
0.92
5.86
0.78
36.25
0.82
24.17
Table 1: Quantitative reconstruction performance across three datasets.
Dataset
CFG
Steps
gFID ↓
gIS ↑
CLIPScore ↑
CUB-200-2011
9.0
16
6.82
5.34
24.74
MS-COCO-2014
2.0
6
7.11
28.67
24.11
Flickr30k
3.0
8
13.86
17.20
23.65
Table 2: Best-performing configurations for each dataset in terms of gFID. More details are provided in Fig. 7 in the Supplement.
Methods
CUB-200-2011
MS-COCO-2014
Steps
gFID ↓
gIS ↑
Steps
gFID ↓
gIS ↑
Improved VQ-Diffusion [ 44 ]
100
10.21
–
100
8.44
–
VQ-Diffusion [ 7 ]
100
10.32
–
100
13.86
–
TCTS+FAS [ 18 ]
16
12.35
–
16
14.45
–
UNIFIED DISCRETE [ 14 ]
500
16.19
6.02
500
25.11
21.03
Ours
4/16
8.03/6.82
5.38/5.34
6
7.11
28.67
Table 3: Comparison with diffusion-based methods trained from scratch.
Figure 4: Qualitative comparison with prior methods using publicly available checkpoints on CUB-200-2011 (left) and MS-COCO-2014 (right).
Methods
CUB-200-2011
MSCOCO2014
gFID ↓
gIS ↑
gFID ↓
gIS ↑
HyperCGAN [ 9 ]
11.00
–
18.58
–
ALR-GAN [ 43 ]
15.14
4.96
29.04
34.70
DFGAN [ 46 ]
14.81
5.10
19.32
–
LAFITE [ 58 ]
10.48
5.97
8.12
32.34
RAT-GAN [ 52 ]
13.91
5.36
14.60
–
Table 4: Comparison with GAN-based methods trained from scratch.
Figure 5: PLS-SVD analysis of the text-conditioned Sphere Encoder with the proposed alignment loss LSSAU on the CUB-200-2011 dataset.
Figure 6: Reconstructed images across three datasets.
Figure 7: Effect of CFG scale and iterative refinement steps on text-conditioned Sphere Encoder across three datasets. Left: gFID ( ↓ ) and gIS ( ↑ ) under varying CFG scales with the number of forward steps fixed at 4. Right: gFID and gIS across different refinement steps (4 − 16, interval 2), using the dataset-specific CFG scales that achieve the lowest gFID, i.e. , best CFG=9.0 on CUB, CFG=2.0 on COCO, CFG=3.0 on Flickr30k.
λu
CUB
COCO
Flickr
rFID
rIS
rFID
rIS
rFID
rIS
0.2
0.92
5.86
0.78
36.25
0.82
24.17
0.1
2.11
5.80
0.89
36.29
0.98
24.12
Table 5: Reconstruction results of rFID and rIS using different uniformity weights across three datasets.
Figure 8: Generation results of gFID and gIS using different uniformity weights across three datasets. w0.2 and w0.1 use uniformity weights of 0.2 and 0.1, respectively. Left: gFID and gIS comparisons under varying CFG scales with the number of forward steps at 4. Right: gFID and gIS across different refinement steps (4-16, interval 2), using the dataset-specific CFG scales that achieve the lowest gFID, i.e. , best CFG=9.0 on CUB, CFG=2.0 on COCO, CFG=3.0 on Flickr30K.
Reconstruction Results (CFG/Step Independent)
Metric
CA
AdaLN
rFID ↓
1.76
0.77
rIS ↑
5.77
5.89
Generation Results (fixed steps=4)
CFG
gFID ↓
gIS ↑
CLIPScore ↑
CA
AdaLN
CA
AdaLN
CA
AdaLN
Table 6: Quantitative comparisons using Cross-Attention (CA) vs. AdaLN for text conditioning on CUB-200-2011 dataset. Top: reconstruction results. Middle: varying CFG scales with fixed forward steps=4. Bottom: varying forward steps with best CFG scale=10.0.
Reconstruction Results (CFG/Step Independent)
Metric
LSphere+λSSAULSSAU (Ours)
LSphere (Original)
rFID ↓
0.92
0.77
rIS ↑
5.86
5.89
Generation Results (fixed steps=4)
CFG
gFID ↓
gIS ↑
CLIPScore ↑
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
Table 7: Ablation study of the proposed SSAU objective on the CUB-200-2011 dataset.
Reconstruction Results (CFG/Step Independent)
Metric
LSphere+λSSAULSSAU (Ours)
LSphere (Original)
rFID ↓
0.78
0.78
rIS ↑
36.25
36.24
Generation Results (fixed steps=4)
CFG
gFID ↓
gIS ↑
CLIPScore ↑
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
Table 8: Ablation study of the proposed SSAU objective on the MS-COCO-2014 dataset.
Reconstruction Results (CFG/Step Independent)
Metric
LSphere+λSSAULSSAU (Ours)
LSphere (Original)
rFID ↓
0.82
0.73
rIS ↑
24.17
24.38
Generation Results (fixed steps=4)
CFG
gFID ↓
gIS ↑
CLIPScore ↑
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
LSphere+λSSAULSSAU
LSphere
Table 9: Ablation study of the proposed SSAU objective on the Flickr30k dataset.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
We introduce the Sphere Encoder, an efficient generative framework capable of producing images in a single forward pass and competing with many-step diffusion models using fewer than five steps. Our approach works by learning an encoder that maps natural images uniformly onto a spherical latent space, and a decoder that maps random latent vectors back to the image space. Trained solely through image reconstruction losses, the model generates an image by simply decoding a random point on the sphere. Our architecture naturally supports conditional generation, and looping the encoder/decoder a few times can further enhance image quality. Across several datasets, the sphere encoder approach yields performance competitive with state of the art diffusions, but with a small fraction of the inference cost. Project page is available at https://sphere-encoder.github.io .
Text-to-image diffusion models are capable of generating high-quality images, but suboptimal pre-trained text representations often result in these images failing to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, CFG introduces significant computational overhead. In this paper, we present DIstilling CFG by sharpening text Embeddings (DICE) that replaces CFG in the sampling process with half the computational complexity while maintaining similar generation quality. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Furthermore, examining the enhancement pattern, we identify the underlying mechanism of DICE that sharpens specific components of text embeddings to preserve semantic information while enhancing fine-grained details. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL, and PixArt-α demonstrate the effectiveness of our method. Code is available at https://github.com/zju-pi/dice.
Zhenyu Zhou, Defang Chen, Can Wang +2
Zhejiang University, State Key Laboratory of Blockchain and Data Security · University at Buffalo, State University of New York
Text-to-image diffusion models can generate visually stunning images, yet, controlling what appears and how it appears, remains surprisingly difficult, especially when operating solely within the constraints of the text-conditioning space. For example, changing a subject or adjusting an attribute often leads to unintended side effects, such as altered backgrounds or distorted details. This is because most existing text-based control methods treat the embedding space as Euclidean and apply simple linear transformations, which do not reflect how semantic concepts are actually organized. In this work, we take a step back and ask: what is the true geometry of these embeddings? We find that text encoder representations lie on a hypersphere, where concepts are not linear directions but structured, anisotropic distributions better captured by Kent distributions. Building on this insight, we propose HEART, a training-free framework that performs Kent-aware geodesic transformations directly on the hypersphere. By respecting the underlying geometry, HEART enables intuitive and precise edits, such as consistent subject replacement and fine-grained attribute control, while preserving the original scene. Importantly, HEART requires no finetuning, inversion, or optimization, and generalizes across diffusion model architectures. Our results show that a simple shift in perspective, from linear to spherical, can unlock fast, and controllable image generation.