Not All Latent Spaces Are Flat: Hyperbolic Concept Control

Maria Rosaria Briglia; Simone Facchiano; Paolo Cursi; Alessio Sampieri; Emanuele Rodolà; Guido Maria D'Amely di Melendugno; Luca Franco; Fabio Galasso

cs.LGMar 14, 2026

Not All Latent Spaces Are Flat: Hyperbolic Concept Control

Authors: Maria Rosaria Briglia, Simone Facchiano, Paolo Cursi, Alessio Sampieri, Emanuele Rodolà, Guido Maria D'Amely di Melendugno, Luca Franco, Fabio Galasso,

Organizations: Department of Computer Science, Sapienza University of Rome, Rome, Italy · ItalAI Labs, Rome, Italy · Paradigma Inc., Rome, Italy

Abstract

As modern text-to-image (T2I) models draw closer to synthesizing highly realistic content, the threat of unsafe content generation grows, and it becomes paramount to exercise control. Existing approaches steer these models by applying Euclidean adjustments to text embeddings, redirecting the generation away from unsafe concepts. In this work, we introduce hyperbolic control (HyCon): a novel control mechanism based on parallel transport that leverages semantically aligned hyperbolic representation space to yield more expressive and stable manipulation of concepts. HyCon reuses off-the-shelf generative models and a state-of-the-art hyperbolic text encoder, linked via a lightweight adapter. HyCon achieves state-of-the-art results across four safety benchmarks and four T2I backbones, showing that hyperbolic steering is a practical and flexible approach for more reliable T2I generation.

Figures & tables

Figure 1: ( top ) In hyperbolic space, concepts (e.g., man or coffee) form entailment cones, and concepts’ composition corresponds to the cones’ intersection. To edit a prompt embedding (e.g., adding coffee to man), we steer it toward the corresponding intersection. ( bottom ) HyCon leverages this hyperbolic geometric structure to add or remove concepts via geometry-consistent edits.

Figure 2: (a) On the COCO training set, we demonstrate that the HyCoCLIP structure effectively maps concept embeddings and their composites into the correct entailment cones, see the discussion in Section 3 . (b) Euclidean ( top ) vs. HyCon ( bottom ) behavior as control strength increases with Stable Diffusion 3.5: Euclidean steering leads to non-smooth or incomplete transitions. By contrast, HyCon follows a smooth geodesic trajectory and remains stable for larger $\lambda$ , consistently increasing the influence of the steered concept.

Figure 3: Semantic alignment distributions of samples retrieved from concept-specific entailment cones. For each concept, the semantic alignment is measured using the CLIPScore of the retrieved embeddings after they are mapped back to their Euclidean representations. Embeddings align more closely with the corresponding concept prompt (blue) than with other concepts (red).

		$C_{\text{Sea}}$	$C_{\text{Grass}}$	$C_{\text{Snow}}$	$C_{\text{Carpet}}$
Caption	R@1	0.00	0.00	0.00	0.00
	R@5	0.04	0.31	0.02	0.14
	R@10	0.07	0.45	0.04	0.23
+ Sea	R@1	0.78	0.10	0.00	0.00
	R@5	0.99	0.36	0.00	0.00
	R@10	0.99	0.56	0.01	0.01

Table 1: Retrieval performance before and after hyperbolic concept control. Captions do not activate any entailment cone

C_{\text{concept}}

. Adding a specific concept to the caption moves embeddings into the intended semantic cone while leaving non-target concepts unaffected, indicating selective and stable steering.

Model	Setting	CLIP $\uparrow$	FID $\downarrow$	Cosine Sim. $\uparrow$
SDXL	Vanilla	0.3264	53.29	0.81
SDXL	Adapted	0.3260	53.40	0.81
SD3	Vanilla	0.3225	58.89	0.82
SD3	Adapted	0.3212	59.79	0.82
SD3.5	Vanilla	0.3250	53.51	0.73
SD3.5	Adapted	0.3232	53.84	0.73

Table 2: Effect of the HyCoCLIP

\rightarrow

CLIP adapter on Flickr30k test set. We compare vanilla CLIP conditioning against adapter-based conditioning across diffusion backbones using CLIPScore, FID, and cosine similarity between vanilla and adapted embeddings.

Method	P4D		Ring-a-Bell		MMA-Diffusion		UnlearnDiffAttk		COCO
	NudeNet $\downarrow$	GPT-4o $\downarrow$	NudeNet $\downarrow$	GPT-4o $\downarrow$	NudeNet $\downarrow$	GPT-4o $\downarrow$	NudeNet $\downarrow$	GPT-4o $\downarrow$	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
SDXL	72.19	41.06	69.62	49.37	35.20	25.00	30.28	9.15	–	0.32	–
SAFREE	26.53	12.93	32.91	17.72	7.80	2.00	12.68	1.41	132.60	0.28	0.77
HyCon	21.19	11.26	26.58	17.72	7.70	0.90	8.45	2.82	60.49	0.27	0.55
SD3	53.64	16.56	67.09	29.11	17.10	9.20	34.51	8.45	–	0.32	–
SAFREE	26.49	3.31	32.91	21.52	9.00	1.40	12.77	2.11	41.82	0.32	0.68

Table 3: Unified evaluation across multiple safety benchmarks and diffusion backbones. Hyperbolic control consistently improves safety-related metrics compared to existing steering methods, while maintaining competitive image quality across datasets.

Figure 4: Qualitative results on Ring-a-Bell (top) and COCO retain set (bottom). For each dataset, columns show Baseline, SAFREE, and HyCon (left to right). On Ring-a-Bell, both methods suppress the target unsafe concept, while on COCO HyCon better preserves non-target content and overall visual fidelity.

Panel of Figure 5 — Figure 5: Qualitative examples. Top row: removing the “Van Gogh” concept HyCon . As the steering strength increases, the generation remains stable and preserves the intended content. Bottom row: adding the concept “night”. Zoom in for details.

Figure 6: Steering–retention trade-offs. Top: NudeNet (Ring-a-Bell) vs. FID (COCO) across steering scales $\lambda$ . Bottom: NudeNet (Ring-a-Bell) vs. CLIP (COCO) across T5 prompt scales.

Method	Ring-a-Bell		COCO
	NudeNet	GPT-4o	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
SDXL	69.62	49.37	–	0.32	–
Euclidean	30.51	19.68	63.87	0.27	0.56
HyCon	26.58	17.72	60.43	0.27	0.55
SD3	67.09	29.11	–	0.32	–
Euclidean	31.65	25.32	39.21	0.31	0.45

Table 4: Safety–retention trade-off for Euclidean refusal-vector steering Vs. HyCon across diffusion backbones. We report nudity suppression on Ring-a-Bell (NudeNet, GPT-4o) and retention on COCO (FID, CLIP, LPIPS).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: This image shows the effect of the growing value of the $\lambda$ parameter on the generation process. The control aim is to insert the concept of ’old’ inside the starting picture. Realized on SD3.5

Figure 8: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to insert is related to the environment and is subtly introduced while keeping the subject unchanged.

Figure 9: The images illustrate the impact of gradually introducing the steering direction shown in the bottom caption. In this case, we want to change the subject of our generation into another one; in this case, the steering primarily affects the subject, keeping the environment unchanged.

Figure 10: The images show the effect, given a fixed $\lambda=3$ , as argued in Sec. 6 , of the removal of the addition of the nudity concept for the SD 3.5 architecture. We can see that the model greatly preserves the context while changing the steered features in both cases.

Figure 11: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to insert is related to the environment or the style of the subject and is subtly introduced while keeping the subject unchanged.

Figure 12: The images illustrate the impact of gradually introducing the steering direction shown in each row. The concept we want to remove is related to the environment or the style of the subject and is subtly introduced while keeping the subject unchanged.

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 215 papers

Introspective Attention Modulation for Safe Text-to-Image Generation
Jul 16, 2026Basim Azam, Hossein Rahmani, Naveed AkhtarDiffusion Transformer T2I Generation
The University of Melbourne, Australia · Lancaster University, United Kingdom
HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models
Sep 25, 2026Yi Sun, Xinhao Zhong, Zhiqi Zhang +3Safe T2I Generation Diffusion Model Unlearning
Harbin Institute of Technology, Shenzhen · Jilin University · Tsinghua Shenzhen International Graduate School, Tsinghua University +1
ForceForget: Reinforcement Concept Removal for Enhancing Safety in Text-to-Image Models
Jun 12, 2026Dong Han, Yong LiSafe T2I Generation AI Safety
Huawei Heisenberg Research Center, Huawei Technologies Düsseldorf GmbH, Düsseldorf, Germany · Computer Vison Group, Friedrich Schiller University Jena, Jena, Germany.

Jul 16, 2026J/K move · Enter open · S save

Introspective Attention Modulation for Safe Text-to-Image Generation

Basim Azam, Hossein Rahmani, Naveed Akhtar

The University of Melbourne, Australia · Lancaster University, United Kingdom

Diffusion Transformer T2I Generation Safe T2I Generation Attention Control in Diffusion Models Controllable Image Generation T2I Diffusion Models+1

Open paper AskPDF arXiv

State-of-the-art flow based text-to-image (T2I) models exhibit remarkable generative abilities but remain vulnerable to producing unsafe content. Prior safety efforts range from concept erasure and prompt filtering to classifier-based gating. However, simple techniques like parameter efficient adaptations of the models easily bypass such guardrails. We introduce a unique principled approach that achieves safety by regulating the model's attention dynamics through inference-time introspection, exhibiting intrinsic robustness. Our method analyzes and rebalances attention activations throughout image synthesis, steering generations away from unsafe concepts while preserving semantic alignment. This introspective control ensures safety of deployed models. Across standard and adversarial safety benchmarks, our approach achieves remarkable safety scores while maintaining or even improving alignment and perceptual quality. Our results reveal that attention-space regulation offers a considerably more promising path to safer diffusion transformer based image generation than the existing concept erasing mechanism.Our code can be accessed at https://basim-azam.github.io/iam/

Abstract

Controllable Image Generation

Explore similar work

Introspective Attention Modulation for Safe Text-to-Image Generation

HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

ForceForget: Reinforcement Concept Removal for Enhancing Safety in Text-to-Image Models