Standard evaluations of backdoor attacks on text-to-image (T2I) models primarily measure trigger activation and visual fidelity. We challenge this paradigm, demonstrating that encoder-side poisoning induces persistent, trigger-free semantic corruption that fundamentally reshapes the representation manifold. We trace this vulnerability to a geometric mechanism: a Jacobian-based analysis reveals that backdoors act as low-rank, target-centered deformations that amplify local sensitivity, causing distortion to propagate coherently across semantic neighborhoods. To rigorously quantify this structural degradation, we introduce SEMAD (Semantic Alignment and Drift), a diagnostic framework that measures both internal embedding drift and downstream functional misalignment. Our findings, validated across diffusion and contrastive paradigms, expose the deep structural risks of encoder poisoning and highlight the necessity of geometric audits beyond simple attack success rates.
Figures & tables
Figure 1: Encoder-level style corruption from backdoor injection. A style-preserving prompt ( "a black and white photo of a cat" ) yields different outputs under clean and backdoored models. (Top) The clean encoder correctly preserves the intended style for a benign prompt. (Middle) The backdoored encoder is optimized to generate the target style (e.g., "bnw") whenever the specific trigger token (e.g., “ó”) is present. (Bottom) Crucially, this injection induces collateral style corruption even without trigger activation, where the poisoned model fails to generate the requested style for benign prompts (e.g., generating color instead of black-and-white).
Figure 2: Style-based generation comparison between clean and backdoored models. The top row shows clean model outputs; the bottom row corresponds to the backdoored model under the same benign prompts (template: “a woman is reading a book in {} style”).
Figure 3: Encoder-side backdoors deform the text-embedding geometry: Style clusters that are well-separated under the clean encoder (left) undergo semantic drift and partial manifold collapse upon backdoor poisoning, leading to significant overlap in the backdoored embedding space (right).
Figure 4: PCA and ECDF analysis of prompt drift under Rickrolling Struppek et al. (2023) . Visualization of Δf(x) for the Rickrolling attack using TAA settings via (a) PCA and (b) ECDF of drift magnitude ∥Δf(x)∥ . Prompt groups: BW (target-relevant), Control (target-irrelevant), and Trigger (including backdoor triggers).
Figure 5: Comparison of Jacobian properties. (a) ECDF of the local sensitivity proxy g(x0) over sampled anchors. Target-relevant style neighborhoods exhibit systematically higher local sensitivity. (b) ECDF of low-rank energy concentration EVR@2 over anchors. Target-relevant anchors show higher concentration.
Figure 6: Semantic drift across prompt groups. ECDF of SDS for (a) Rickrolling and (b) Nightshade over three groups: trigger-containing prompts (Trigger(ó)/Trigger( Dog )), target-relevant prompts (BW/Cat), and matched target-irrelevant prompts(Control).
Figure 7: Violin plots under different Rickrolling attacks. We compare the distributions of semantic drift score (SDS) between target-relevant prompts ( Target ) and target-irrelevant prompts ( Control ) across different attack variants. Numbers above each panel denote the mean SDS for Target (left) and Control (right).
Figure 8: ECDF of CLIP similarity deltas Δs(x)=s(x,Ibd)−s(x,Iclean) under Rickrolling and Nightshade Attacks. BW and Cat ( target-relevant ) prompt sets exhibit a clear left shift under attacks. Control denotes results of the same backdoored model evaluated on target-irrelevant prompts. Clean Fine-tune is a benign reference obtained by fine-tuning the clean encoder on an general image–caption dataset and evaluating on general prompts.
Figure 9: ECDF of SDS for three prompt groups: the target concept (Lorikeet), semantic neighbors (Bird), and target-irrelevant controls. Embeddings are Procrustes-aligned on unrelated controls.
Figure 10: PCA analysis of encoder embeddings under Noisy Alignment. Comparison of (a) clean and (b) poisoned encoders. Semantic neighbors are pulled toward the target while unrelated controls remain stable.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Rickrolling(BW): KDE of CLIP score deltas show a clear leftward shift for BW-sensitive prompts compared to control prompts, indicating systematic semantic degradation.
Figure 12: Textual Inversion (BW): KDE of CLIP -score deltas Δs for BW-relevant vs. control prompts.
Figure 13: Textual Inversion(BW): ECDF of CLIP -score deltas Δs for BW-relevant prompts.
Quantile (%)
Relevant Prompts
Irrelevant Prompts
10
−0.0185
−0.0126
5
−0.0261
−0.0229
1
−0.0415
−0.0423
Appendix
Table 1: Δ CLIP similarity quantiles. The CLIP-based similarity deltas Δs(x)=s(x,Ibd)−s(x,Iclean) denote the alignment shift, where negative values quantify the magnitude of semantic degradation.
Figure 14: Layer-wise PCA of encoder perturbations for a style-based backdoor (BW).
Figure 15: Layer-wise PCA of encoder perturbations for a object-based backdoor (Dog). Compared to BW style attacks, variance is distributed across multiple components, indicating a higher-rank and more diffuse perturbation structure.