Breaking Visual Conventions: Spectrally Coupled Weight Perturbation for Creative Image Generation
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Abstract
Text-to-image (T2I) models often produce high-quality images that follow familiar visual conventions, even in response to open-ended prompts such as ``a creative [CLS]''. Existing methods encourage novelty by exploring semantic conditions or modifying features and guidance during denoising. Pretrained weights offer a complementary source of creative variation, but unstructured perturbations can have little visual effect or degrade image quality. We introduce Coupled Spectral Perturbation (CosPert), a training-free method that targets attention projections in layers identified by a one-time sensitivity analysis. For each projection, CosPert uses a shared random transformation and its transpose to couple updates to the singular-vector factors while perturbing the spectral gains. A common relative budget calibrates the magnitude of weight changes across projections, and a sigmoid schedule concentrates their influence in early denoising and gradually attenuates them during refinement. CosPert provides plug-and-play creativity enhancement at inference time. Once the perturbations are prepared, sampling requires no auxiliary networks or additional model evaluations. On both FLUX.1 and Stable Diffusion 3.5, CosPert enhances visual creativity, producing more varied object structures while preserving category identity. Across 50 object categories, CosPert achieves the highest mean CREward among the evaluated methods while maintaining strong prompt alignment.
Figures & tables
| FLUX.1 | SD3.5 | GPT | Nano Banana | C3 | VLM-Guide | SCDiff | CREA | CosPert | ||
| (a) | CREward | |||||||||
| (b) | VQAScore | |||||||||
| PickScore | ||||||||||
| (c) | User Study |
T2I Generation
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
See all 329 papersConcept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
A Creative Agent is Worth a 64-Token Template
a creative vinyl record-inspired skyscraper'', these models often fail to infer the underlying creative intent, leaving creative ideation and prompt design largely to human users. Recent reasoning- or agent-driven approaches iteratively augment prompts but incur high computational and monetary costs, as their instance-specific generation makes creativity'' costly and non-reusable, requiring repeated queries or reasoning for subsequent generations. To address this, we introduce CAT, a framework for Creative Agent Tokenization that encapsulates agents' intrinsic understanding of ``creativity'' through a Creative Tokenizer. Given the embeddings of fuzzy prompts, the tokenizer generates a reusable token template that can be directly concatenated with them to inject creative semantics into T2I models without repeated reasoning or prompt augmentation. To enable this, the tokenizer is trained via creative semantic disentanglement, leveraging relations among partially overlapping concept pairs to capture the agent's latent creative representations. Extensive experiments on Architecture Design, Furniture Design, and Nature Mixture tasks demonstrate that CAT provides a scalable and effective paradigm for enhancing creativity in T2I generation, achieving a speedup and a reduction in computational cost, while producing images with superior human preference and text-image alignment compared to state-of-the-art T2I models and creative generation methods.