As modern text-to-image (T2I) models draw closer to synthesizing highly realistic content, the threat of unsafe content generation grows, and it becomes paramount to exercise control. Existing approaches steer these models by applying Euclidean adjustments to text embeddings, redirecting the generation away from unsafe concepts. In this work, we introduce hyperbolic control (HyCon): a novel control mechanism based on parallel transport that leverages semantically aligned hyperbolic representation space to yield more expressive and stable manipulation of concepts. HyCon reuses off-the-shelf generative models and a state-of-the-art hyperbolic text encoder, linked via a lightweight adapter. HyCon achieves state-of-the-art results across four safety benchmarks and four T2I backbones, showing that hyperbolic steering is a practical and flexible approach for more reliable T2I generation.
Figures & tables
Figure 1: ( top ) In hyperbolic space, concepts (e.g., man or coffee) form entailment cones, and concepts’ composition corresponds to the cones’ intersection. To edit a prompt embedding (e.g., adding coffee to man), we steer it toward the corresponding intersection. ( bottom ) HyCon leverages this hyperbolic geometric structure to add or remove concepts via geometry-consistent edits.
Figure 2: (a) On the COCO training set, we demonstrate that the HyCoCLIP structure effectively maps concept embeddings and their composites into the correct entailment cones, see the discussion in Section 3 . (b) Euclidean ( top ) vs. HyCon ( bottom ) behavior as control strength increases with Stable Diffusion 3.5: Euclidean steering leads to non-smooth or incomplete transitions. By contrast, HyCon follows a smooth geodesic trajectory and remains stable for larger λ , consistently increasing the influence of the steered concept.
Figure 3: Semantic alignment distributions of samples retrieved from concept-specific entailment cones. For each concept, the semantic alignment is measured using the CLIPScore of the retrieved embeddings after they are mapped back to their Euclidean representations. Embeddings align more closely with the corresponding concept prompt (blue) than with other concepts (red).
CSea
CGrass
CSnow
CCarpet
Caption
R@1
0.00
0.00
0.00
0.00
R@5
0.04
0.31
0.02
0.14
R@10
0.07
0.45
0.04
0.23
+ Sea
R@1
0.78
0.10
0.00
0.00
R@5
0.99
0.36
0.00
0.00
R@10
0.99
0.56
0.01
0.01
Table 1: Retrieval performance before and after hyperbolic concept control. Captions do not activate any entailment cone Cconcept . Adding a specific concept to the caption moves embeddings into the intended semantic cone while leaving non-target concepts unaffected, indicating selective and stable steering.
Model
Setting
CLIP ↑
FID ↓
Cosine Sim. ↑
SDXL
Vanilla
0.3264
53.29
0.81
Adapted
0.3260
53.40
SD3
Vanilla
0.3225
58.89
0.82
Adapted
0.3212
59.79
SD3.5
Vanilla
0.3250
53.51
0.73
Adapted
0.3232
53.84
Table 2: Effect of the HyCoCLIP → CLIP adapter on Flickr30k test set. We compare vanilla CLIP conditioning against adapter-based conditioning across diffusion backbones using CLIPScore, FID, and cosine similarity between vanilla and adapted embeddings.
Method
P4D
Ring-a-Bell
MMA-Diffusion
UnlearnDiffAttk
COCO
NudeNet ↓
GPT-4o ↓
NudeNet ↓
GPT-4o ↓
NudeNet ↓
GPT-4o ↓
NudeNet ↓
GPT-4o ↓
FID ↓
CLIP ↑
LPIPS ↓
SDXL
72.19
41.06
69.62
49.37
35.20
25.00
30.28
9.15
–
0.32
–
SAFREE
26.53
12.93
32.91
17.72
7.80
2.00
12.68
1.41
132.60
0.28
0.77
HyCon
21.19
11.26
26.58
17.72
7.70
0.90
8.45
2.82
60.49
0.27
0.55
SD3
53.64
16.56
67.09
29.11
17.10
9.20
34.51
8.45
–
0.32
–
SAFREE
26.49
3.31
32.91
21.52
9.00
1.40
12.77
2.11
41.82
0.32
0.68
Table 3: Unified evaluation across multiple safety benchmarks and diffusion backbones. Hyperbolic control consistently improves safety-related metrics compared to existing steering methods, while maintaining competitive image quality across datasets.
Figure 4: Qualitative results on Ring-a-Bell (top) and COCO retain set (bottom). For each dataset, columns show Baseline, SAFREE, and HyCon (left to right). On Ring-a-Bell, both methods suppress the target unsafe concept, while on COCO HyCon better preserves non-target content and overall visual fidelity.
Figure 5: Qualitative examples. Top row: removing the “Van Gogh” concept HyCon . As the steering strength increases, the generation remains stable and preserves the intended content. Bottom row: adding the concept “night”. Zoom in for details.
Figure 9
Figure 6: Steering–retention trade-offs. Top: NudeNet (Ring-a-Bell) vs. FID (COCO) across steering scales λ . Bottom: NudeNet (Ring-a-Bell) vs. CLIP (COCO) across T5 prompt scales.
Method
Ring-a-Bell
COCO
NudeNet
GPT-4o
FID ↓
CLIP ↑
LPIPS ↓
SDXL
69.62
49.37
–
0.32
–
Euclidean
30.51
19.68
63.87
0.27
0.56
HyCon
26.58
17.72
60.43
0.27
0.55
SD3
67.09
29.11
–
0.32
–
Euclidean
31.65
25.32
39.21
0.31
0.45
Table 4: Safety–retention trade-off for Euclidean refusal-vector steering Vs. HyCon across diffusion backbones. We report nudity suppression on Ring-a-Bell (NudeNet, GPT-4o) and retention on COCO (FID, CLIP, LPIPS).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: This image shows the effect of the growing value of the λ parameter on the generation process. The control aim is to insert the concept of ’old’ inside the starting picture. Realized on SD3.5
Figure 8: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to insert is related to the environment and is subtly introduced while keeping the subject unchanged.
Figure 9: The images illustrate the impact of gradually introducing the steering direction shown in the bottom caption. In this case, we want to change the subject of our generation into another one; in this case, the steering primarily affects the subject, keeping the environment unchanged.
Figure 10: The images show the effect, given a fixed λ=3 , as argued in Sec. 6 , of the removal of the addition of the nudity concept for the SD 3.5 architecture. We can see that the model greatly preserves the context while changing the steered features in both cases.
Figure 11: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to insert is related to the environment or the style of the subject and is subtly introduced while keeping the subject unchanged.
Figure 12: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to remove is related to the environment or the style of the subject and is subtly introduced while keeping the subject unchanged.
State-of-the-art flow based text-to-image (T2I) models exhibit remarkable generative abilities but remain vulnerable to producing unsafe content. Prior safety efforts range from concept erasure and prompt filtering to classifier-based gating. However, simple techniques like parameter efficient adaptations of the models easily bypass such guardrails. We introduce a unique principled approach that achieves safety by regulating the model's attention dynamics through inference-time introspection, exhibiting intrinsic robustness. Our method analyzes and rebalances attention activations throughout image synthesis, steering generations away from unsafe concepts while preserving semantic alignment. This introspective control ensures safety of deployed models. Across standard and adversarial safety benchmarks, our approach achieves remarkable safety scores while maintaining or even improving alignment and perceptual quality. Our results reveal that attention-space regulation offers a considerably more promising path to safer diffusion transformer based image generation than the existing concept erasing mechanism.Our code can be accessed at https://basim-azam.github.io/iam/
Basim Azam, Hossein Rahmani, Naveed Akhtar
The University of Melbourne, Australia · Lancaster University, United Kingdom
Recent advances in text-to-image (T2I) generation have substantially improved visual synthesis, but have also raised increasing safety concerns due to their potential to generate harmful or undesirable content. Existing concept erasure methods predominantly follow a static weight paradigm, producing a single frozen adapter that struggles to adapt to diverse prompt variations and suffers from parameter interference when scaling to multiple concepts. We propose HyperErase, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. Our approach first reframes concept erasure as prompt-conditioned parameter amortization and trains a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt gradient optimization or manual LoRA merging. To further improve the stability and precision of synthesized adapters, we develop a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction. Extensive experiments across major concept categories demonstrate that HyperErase consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment, achieving performance comparable to gold-standard single-concept baselines. Furthermore, the resulting models can provide specialized LoRAs for each input prompt variation in a single forward pass without requiring gradient updates during inference. These principled and flexible framework offers a new paradigm for concept erasure in T2I models.
Yi Sun, Xinhao Zhong, Zhiqi Zhang +3
Harbin Institute of Technology, Shenzhen · Jilin University · Tsinghua Shenzhen International Graduate School, Tsinghua University +1
With the advance of generative AI, the text-to-image (T2I) model has the ability to generate various contents. However, T2I models still can generate unsafe contents. To alleviate this issue, various concept erasing methods are proposed. However, existing methods tend to excessively erase unsafe concepts and suppress benign concepts contained in harmful prompts, which can negatively affect model utility. In this paper, we focus on eliminating unsafe content while maintaining model capability in safe semantic meaning interpretation by optimizing the concept erasing reward (CER) with reinforcement learning. To avoid overly content erasure, we introduce the Safe Adapter to project partial text embedding for efficient concept regulation in cross-attention layers. Extensive experiments conducted on different datasets demonstrate the effectiveness of the proposed method in alleviating unsafe content generation while preserving the high fidelity of benign images compared with existing state-of-the-art (SOTA) concept erasing methods. In terms of robustness, our method outperforms counterparts against red-teaming tools. Moreover, we showcase the proposed approach is more effective in emerging image-to-image (I2I) scenarios compared with others. Lastly, we extend our method to erase general concepts, such as artistic styles and objects. Disclaimer: This paper includes discussions of sexually explicit content that may be offensive to certain readers. All images used in this work are synthesized or from public datasets.
Dong Han, Yong Li
Huawei Heisenberg Research Center, Huawei Technologies Düsseldorf GmbH, Düsseldorf, Germany · Computer Vison Group, Friedrich Schiller University Jena, Jena, Germany.