CTCal: Rethinking Text-to-Image Diffusion Models via Cross-Timestep Self-Calibration

Xiefan Guo; Xinzhu Ma; Haiyu Zhang; Di Huang

cs.CVMar 21, 2026

CTCal: Rethinking Text-to-Image Diffusion Models via Cross-Timestep Self-Calibration

Authors: Xiefan Guo, Xinzhu Ma, Haiyu Zhang, Di Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China

Abstract

Recent advancements in text-to-image synthesis have been largely propelled by diffusion-based models, yet achieving precise alignment between text prompts and generated images remains a persistent challenge. We find that this difficulty arises primarily from the limitations of conventional diffusion loss, which provides only implicit supervision for modeling fine-grained text-image correspondence. In this paper, we introduce Cross-Timestep Self-Calibration (CTCal), founded on the supporting observation that establishing accurate text-image alignment within diffusion models becomes progressively more difficult as the timestep increases. CTCal leverages the reliable text-image alignment (i.e., cross-attention maps) formed at smaller timesteps with less noise to calibrate the representation learning at larger timesteps with more noise, thereby providing explicit supervision during training. We further propose a timestep-aware adaptive weighting to achieve a harmonious integration of CTCal and diffusion loss. CTCal is model-agnostic and can be seamlessly integrated into existing text-to-image diffusion models, encompassing both diffusion-based (e.g., SD 2.1) and flow-based approaches (e.g., SD 3). Extensive experiments on T2I-Compbench++ and GenEval benchmarks demonstrate the effectiveness and generalizability of the proposed CTCal. Our code is available at https://github.com/xiefan-guo/ctcal.

Figures & tables

Figure 1: Investigation on the cross-attention maps. (a) Inference stage. In line with existing inference-time optimization methods [ 15 , 5 ] , we delve into the analysis of cross-attention maps produced during the inference stage of the text-to-image diffusion model. Notably, satisfactory text-image correspondences are established for simple text prompts. Nevertheless, with more intricate text prompts, the prevalent method encounters challenges in precisely mapping the target semantics to the correct spatial position, leading to semantically inconsistent images. (b) Training stage. Given the text-image-noise triplet, we gather cross-attention maps at varied timesteps in training mode. A noteworthy finding emerges: cross-attention maps obtained at smaller timesteps exhibit substantially better alignment with the ground-truth image structure and semantics, while this alignment substantially deteriorates at larger timesteps. This suggests that the conventional diffusion loss, which is ubiquitously employed in current training protocols, is effective primarily at smaller timesteps. Moreover, this inability to establish precise alignments at larger timesteps, i.e. , initial stage of inference process, constitutes a critical bottleneck, fundamentally constraining the overall fidelity and semantic accuracy of text-to-image generation.

Figure 2: Illustration of CTCal . CTCal is dedicated to leverage the reliable text-image alignment established at smaller timesteps ( $t_{\text{tea}}$ ) to calibrate the learning process at larger timesteps ( $t_{\text{stu}}$ ). This approach provides explicit supervision for the modeling of text-image correspondence, thereby enhancing the overall performance of text-to-image generation. Notably, the two diffusion models share identical parameters, which is solely for the convenience of presentation.

Figure 3: Investigation on cross-attention maps categorized by part-of-speech. Cross-ttention maps for noun tokens ( e.g. , “cat”) typically encode clear spatial semantic information, while those for articles ( e.g. , “the”) and conjunctions ( e.g. , “and”) seemingly lack a significant conveyance.

Panel of Figure 4 — Figure 4: Qualitative comparison on SD 2.1 and SD 3. CTCal demonstrates a marked improvement in the fine-grained alignment of generated images with the corresponding text prompts. Each image is generated with the same prompt and random seed for all methods.

Methods	Color B-VQA	Shape B-VQA	Texture B-VQA	2D-Spatial UniDet	3D-Spatial UniDet	Numeracy UniDet	Non-Spatial Share-CoT	Complex 3-in-1
SD 1.4 [ 32 ]	0.3765	0.3576	0.4156	0.1246	0.3030	0.4456	0.7487	0.3080
SD 2.1 [ 32 ]	0.5065	0.4221	0.4922	0.1342	0.3230	0.4582	0.7567	0.3386
SD 2.1 + CD [ 23 ]	0.4063	0.3299	0.3645	0.0800	0.2847	0.4272	0.6927	0.2898
SD 2.1 + SD [ 10 ]	0.4990	0.4218	0.4900	0.1386	0.3224	0.4557	0.7560	0.3355
SD 2.1 + AE [ 5 ]	0.6400	0.4517	0.5963	0.1455	0.3222	0.4773	0.7593	0.3401
SD 2.1 + GORS 1	0.6603	0.4785	0.6287	0.1815	0.3572	0.4830	0.7637	0.3328

Table 1: Objective evaluation on T2I-CompBench++. Top: conventional text-to-image diffusion models, inference-time optimization approaches, supervised fine-tuning methods, and our method. Bottom: advanced text-to-image diffusion models, supervised fine-tuning methods, and our method. CTCal exhibits the superior performance in attribute binding, object relationships, counting, and complex compositions, highlighting the advanced capability for compositional generation.

Table 6

Methods	Color B-VQA	2D-Spatial UniDet
SD 2.1	0.5065	0.1342
+ GORS (baseline)	0.6426	0.1775
+ CTCal ( a )	0.6286 ( -2.18% )	0.1693 ( -4.62% )
+ CTCal ( b )	0.6897 ( +7.33% )	0.1972 ( +11.10% )
+ CTCal ( c )	0.6992 ( +8.81% )	0.2021 ( +13.86% )
+ CTCal ( d )	0.7148 ( +11.24% )	0.2095 ( +18.03% )

Table 4: Ablation study on Stable Diffusion 2.1.

Methods	Color B-VQA	Texture B-VQA
SD 2.1	0.5065	0.4922
+ GORS	0.6426	0.6319
+ CTCal	0.7233	0.6754
+ CTCal + adj. token	0.7328	0.6877

Table 6: Objective evaluation on adj. .

Methods	Color M-LPIPS	2D-Spatial M-LPIPS	Color Aesthetic	2D-Spatial Aesthetic
SD 2.1	0.637	0.618	5.128	5.263
+ GORS	0.621	0.626	5.194	5.281
+ CTCal	0.634	0.623	5.288	5.344

Table 7: More results on diversity and quality evaluation.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 403 papers

AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis
May 25, 2026Shipeng Cao, Biao Qian, Haipeng Liu +2Diffusion Model Alignment Cross-Attention
Key Laboratory of Knowledge Engineering With Big Data, Ministry of Education, Hefei University of Technology, China · Department of Automation, Tsinghua University, China
ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models
Jun 11, 2025Qin Zhou, Zhiyang Zhang, Jinglong Wang +5Diffusion Model Alignment Text-Image Alignment
School of Software, Beihang University, Beijing 100191, China · College of Computer and Cyber Security, Hebei Normal University, Hebei 050024, China · Department of Computer Science, The University of Hong Kong, Hong Kong
RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment
Feb 28, 2026Liyao Jiang, Ruichen Chen, Chao Gao +1T2I Generation Text-Image Alignment
Department of ECE, University of Alberta, Canada · Huawei Technologies, Canada

May 25, 2026J/K move · Enter open · S save

AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis

Shipeng Cao, Biao Qian, Haipeng Liu, Yang Wang+1

Key Laboratory of Knowledge Engineering With Big Data, Ministry of Education, Hefei University of Technology, China · Department of Automation, Tsinghua University, China

Diffusion Model Alignment Cross-Attention Attention Control in Diffusion Models Compositional T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image synthesis has made significant progress, benefiting from the strong generative capabilities of diffusion models. However, these models struggle to achieve precise text-to-image alignment within cross-attention maps during the denoising process. Existing works primarily focus on inter-subject-token activations (i.e., cross-attention scores) overlap for different subjects, overlooking the intra-subject-token activations scattering issue for identical subjects. In this paper, we propose an Aggregating-and-Isolating cross-attention approach to diffusion models for Text-to-Image synthesis, dubbed AI-T2I. Technically, to address the scattering issue, we devise an aggregation loss to identify and consolidate the scattered intra-token activations, which implicitly helps mitigate the potential overlap issue. Upon that, an isolation loss is further introduced to push the inter-token activations apart, thus fulfilling precise text-to-image alignment. Extensive experiments on various benchmarks demonstrate the superiority of AI-T2I over the state-of-the-art works for text-to-image synthesis. Furthermore, our AI-T2I exhibits excellent generalization across other tasks, e.g., controllable layout generation and personalized generation. Our code is available at https://github.com/Hatter77/AI-T2I.

Abstract

T2I Diffusion Models

Explore similar work

AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis

ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models

RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment