ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference
Organizations: School of Computing and Information Systems, The University of Melbourne
Abstract
Text-to-image models have achieved remarkable success in generating high-quality images from textual descriptions. However, they often struggle with "semantic drift," where the generated output fails to precisely align with complex or nuanced text prompts. While recent approaches have attempted to address semantic errors regarding attribute binding or object presence, there remains a gap for a more holistic method that addresses these issues by directly refining the text embeddings of the initial user prompt. In this work, we introduce Orthogonal Drift Correction (ODC), an inference-time guidance method designed to mitigate semantic drift without requiring model retraining or additional user inputs. ODC guides image generation through a two-stage process. In the first stage, it identifies the semantic drift by evaluating the initially generated image against the user prompt in a shared vision-language embedding space. It then isolates the component of this drift vector that is orthogonal to the prompt's direction and translates it back into text via a vocabulary-based surrogate mechanism. In the second stage, it produces refined text conditioning for a second generation pass by feeding both the initial text embedding and the re-embedded drift representation into an adaptive rank-reduced concept removal module. Our experiments demonstrate that ODC consistently improves prompt-image alignment, increasing CLIP scores by up to 5.4% over the baseline on the COALBench-300 and PartiPrompts benchmarks. Furthermore, on the T2I-CompBench complex subset, ODC improves the average 3-in-1 score by 4.5% over the competing methods. As a plug-and-play module, ODC offers a practical and efficient method for improving the reliability of text-to-image models. Code is available at https://github.com/kasrakamali/ODC.
Figures & tables
| Method | Coal Bench-300 | PartiPrompts | ||||||
|---|---|---|---|---|---|---|---|---|
| SDXL | FLUX.1 [schnell] | SDXL | FLUX.1 [schnell] | |||||
| CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | |
| Vanilla | 30.3279 | 0.6078 | 29.4499 | 0.6076 | 30.8338 | 0.6030 | 31.2665 | 0.6051 |
| CFG-Sweep (CFG=5.0) | 30.0022 | 0.6070 | - | - | 30.4468 | 0.6021 | - | - |
| CFG-Sweep (CFG=12.0) | 30.4509 | 0.6083 | - | - | 31.1269 | 0.6040 | - | - |
| Negative Prompt | 30.3282 | 0.6078 | 29.4616 | 0.6074 | 30.8336 | 0.6030 | 31.1844 | 0.6051 |
| Method | Coal Bench-300 | PartiPrompts | T2I-CompBench (Complex) | |||||
|---|---|---|---|---|---|---|---|---|
| CLIP | BLIP | CLIP | BLIP | BLIP-VQA | UniDet | CLIP | 3-in-1 | |
| SDXL ( Podell et al., 2023 ) | 30.305 | 0.608 | 30.834 | 0.603 | 0.442 | 0.025 | 0.293 | 0.3294 |
| A&E ( Chefer et al., 2023 ) | 29.542 | 0.608 | 30.15 | 0.604 | 0.436 | 0.023 | 0.290 | 0.3259 |
| ToMe ( Hu et al., 2024 ) | 30.397 | 0.608 | 30.681 | 0.603 | 0.468 | 0.022 | 0.292 | 0.3417 |
| ODC (Ours) | 31.741 | 0.612 | 32.503 | 0.607 | 0.494 | 0.020 | 0.300 | 0.3572 |
| Method | Latency (s/img) | Memory (GB) |
|---|---|---|
| Vanilla | 3.637 | 23.015 |
| ODC | 7.560 | 23.015 |
| Metric | Value | 95% CI |
|---|---|---|
| ODC Win Rate | 61.8% | [57.0%, 66.3%] |
| Baseline Win Rate | 38.2% | [33.7%, 43.0%] |
| Tie Rate | 30.7% | — |
| CLIP ViT-L/14 | SigLIP2 SO400M | EVA-Giant Plus | BLIP2-ITM-ViT-G | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | FLUX.1 | SDXL | FLUX.1 | SDXL | FLUX.1 | SDXL | FLUX.1 | |||||||||
| Method | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP | CLIP | BLIP |
| Vanilla | 30.258 | 0.607 | 29.329 | 0.607 | 30.258 | 0.607 | 29.329 | 0.607 | 30.258 | 0.607 | 29.329 | 0.607 | 30.258 | 0.607 | 29.329 | 0.607 |
| ODC | 31.662 | 0.611 | 29.616 | 0.608 | 31.173 | 0.610 | 29.504 | 0.608 | 31.642 | 0.611 | 29.629 | 0.608 | 31.762 | 0.612 | 29.778 | 0.608 |
| ODC BoT | 32.106 | 0.612 | 30.105 | 0.609 | 31.559 | 0.611 | 30.141 | 0.609 | 32.126 | 0.612 | 30.354 | 0.610 | 32.206 | 0.612 | 30.278 | 0.610 |
| Method | CLIP |
|---|---|
| Vanilla | 30.305 |
| Simple Concept Removal | 30.970 |
| Full Vector Correction | 31.489 |
| Orthogonal Drift Correction | 31.741 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Prompt | # | Prompt |
|---|---|---|---|
| 1 | Neon cityscape reflected in rain puddles at midnight | 2 | Ancient library filled with floating glowing books |
| 3 | Steampunk octopus playing violin underwater | 4 | Cherry blossoms falling on samurai armor |
| 5 | Cyberpunk cat cafe with holographic menu displays | 6 | Dragon made entirely of autumn leaves |
| 7 | Astronaut planting flowers on Mars surface | 8 | Victorian greenhouse overflowing with bioluminescent plants |
| 9 | Robot chef preparing sushi in minimalist kitchen | 10 | Northern lights dancing above crystal ice castle |
| 11 | Abandoned amusement park reclaimed by nature | 12 | Phoenix rising from coffee cup steam |
| Model | Metric | Baseline (Mean Std) | ODC (Mean Std) | ODC BoT (Mean Std) | (95% CI) | -value |
|---|---|---|---|---|---|---|
| SDXL | CLIP | 30.328 3.226 | 31.798 3.213 | 32.291 3.043 | 1.470 [1.165, 1.774] | |
| BLIP | 0.6078 0.0074 | 0.6121 0.0067 | 0.6127 0.0064 | 0.0042 [0.0036, 0.0048] | ||
| FLUX.1 | CLIP | 29.450 3.386 | 29.918 3.224 | 30.455 3.227 | 0.468 [0.239, 0.698] | |
| BLIP | 0.6076 0.0072 | 0.6084 0.0071 | 0.6097 0.0069 | 0.0008 [0.0004, 0.0012] |
| Index | Prompt Text | Model | Retrieved Drift Words ( ) | Interpretation |
|---|---|---|---|---|
| 151 | A tree whose leaves are made of delicate, colored glass | SDXL | stained glass art , glass art , light panels , light panel , glasswork | Unintended glass/art style elements. |
| 149 | A medieval feast with ghosts sitting amongst the living guests | SDXL | the abbess , illustre , preillustration , overillustration , tabernacler | Misplaced religious/illustrative terms. |
| 30 | Paint brushes growing like flowers in pot | SDXL | utensils or brushes , craft materials , brush land , petrographer , makeup kit | Spurious, unrelated tools/materials. |
| 4 | Cherry blossoms falling on samurai armor | SDXL | wallpaper , rider and horse , person horse , illustre , art wall decor | Misplaced compositional elements. |
| 46 | Giant mushroom houses in enchanted forest | SDXL | mushroom by , forest floor , agaricic , agaricales , toader | Unexpected botanical elements. |
| 63 | Clockwork owl perched on moon | SDXL | tartary , overillustration , odontolith , nightfowl , antecavern | Unrelated archaic/anatomical terms. |
Prompt Optimization for T2I Generation
Prompt optimization for text-to-image generation improves how natural-language instructions control image content, style, composition, and quality.
See all 34 papers