From Concept Erasure to Style Purification: Contrastive Eigenbases for Artist Style Protection

Tong Zhang; Ru Zhang; Jianyi Liu

cs.CVFeb 8, 2026

From Concept Erasure to Style Purification: Contrastive Eigenbases for Artist Style Protection

Authors: Tong Zhang, Ru Zhang, Jianyi Liu

Organizations: Beijing University of Posts and Telecommunications, School of Cyberspace Security, Beijing, 100876

Abstract

Text-to-image diffusion models can reproduce specific artists visual styles at extremely low cost, raising copyright and deployment safety concerns about unauthorized style mimicry. Existing model-side protection methods generally follow ordinary concept erasure, emphasizing aggressive deletion or redirection of target styles. However, our causal intervention analysis shows that the central issue is not insufficient erasure strength, but a mismatch between artist styles and this paradigm: unlike ordinary object concepts, artist styles do not form compact, localized editable semantic units. Consequently, sparse editing and fixed retain lists struggle to suppress target styles while preserving generation utility. We therefore reformulate artist style protection as style purification, suppressing target style expression during inference while preserving the requested content and visual structure. We propose CAPE (Contrastive Artist Style Purification with Eigenbases), a training-free framework against artist style mimicry. CAPE constructs contrastive triplets around the target request and formulates style direction estimation as a generalized eigenvalue problem, capturing style-related directions that remain stable across content variations and are less affected by shared content. During inference, CAPE employs the Adaptive Suppression Controller to assign suppression strengths to different tokens based on Q, K, and V responses, and performs target style suppression on the K and V paths of self-attention. Experimental results show that CAPE effectively weakens target artist characteristics, including brushstrokes, textures, and local color processing, while better preserving major semantic entities, scene composition, and visual structures.

Figures & tables

Figure 1 : Component level causal intervention for erasure analysis. Starting from the original model, we inject only one cross attention layer component from the erased checkpoint at each time while keeping the remaining components unchanged. The causal contribution of each component is quantified by evaluating the difference between the generated images before and after intervention.

Metric	Artist style	Object	Celebrity
$N(c)$	0.702	0.635	0.229
$M_{c}(1)$	0.209	0.262	0.724
$M_{c}(3)$	0.484	0.539	0.836
$M_{c}(5)$	0.658	0.700	0.893

Table 1 : Distributed causal support of erasure across different concept categories. Artist styles exhibit higher causal dispersion and lower Top-

k

contribution mass compared with object and celebrity concepts.

Figure 2 : (a) Box plots of causal component dispersion $N(c)$ for different concept categories. Compared with other concept categories, artist style exhibits more distributed effective causal support. (b) Cumulative target suppression under increasing cross attention coverage. Artist style suppression accumulates more slowly under sparse intervention and gradually approaches the full edit effect only with broader component coverage.

Metric	$D^{content,CSD}$		$D^{content,VGG}$
	Style	Object	Style	Object
$D_{near}$	0.345	0.394	0.343	0.407
$D_{far}$	0.325	0.294	0.337	0.274
$D_{near}^{Top8}$	0.370	0.465	0.365	0.480
$D_{far}^{Top8}$	0.323	0.248	0.328	0.233

Table 2 : Non-target collateral footprint under CSD-based and VGG-based damage measurements.

Figure 3 : Overview of CAPE. The first stage constructs contrastive triplets and estimates $U_{style}$ by solving a generalized eigenvalue problem. During generation, ASC estimates token level style concentration from Q, K, and V responses and adaptively suppresses the projected style associated responses in the K and V paths of self-attention.

Figure 4 : Qualitative comparison with representative baselines on Van Gogh, Maurice De Vlaminck, Adrian Ghenie, and Charles Camoin. CAPE suppresses characteristic style patterns while preserving the requested content and scene structure more consistently across diverse examples.

Method	$C_{VGG}\uparrow$	$C_{\mathrm{style}}\uparrow$	$C_{CSD}\uparrow$	$S_{sup}\uparrow$	CLIP Acc $\uparrow$
SPEED [ 15 ]	0.0566	0.0485	0.1368	2.55	0.9587
ESD-u [ 8 ]	0.1148	0.0747	0.2346	2.67	0.9365
ESD-x [ 8 ]	0.0971	0.1005	0.3657	4.30	0.9341
FMN [ 35 ]	0.0871	0.0653	0.1790	2.65	0.9675
MACE [ 16 ]	0.1077	0.0907	0.2744	4.88	0.9730
P2P [ 12 ]	0.0115	0.0195	0.0596	0.66	0.9000

Table 3 : Standard performance evaluation results. CAPE effectively suppresses target style characteristics while preserving semantic content more consistently than the compared methods.

Figure 5 : Qualitative comparison with prompt filtering under explicit, implicit, and adversarial requests for Van Gogh style. From top to bottom, they are: original, Filtered, CAPE. For each content scene, the three prompt variants target the same artist style while using different textual expressions. Prompt filtering fails under implicit or obfuscated artist cues, whereas CAPE suppresses target-style patterns across all three prompt forms.

Estimation	$H_{\mathrm{style}}\uparrow$	$H_{\mathrm{content}}\downarrow$	$S_{sup}\uparrow$	$cs_{\mathrm{style}}\downarrow$	$cs_{\mathrm{content}}\uparrow$
Random	6.643	6.654	1.7	24.29	30.83
Shuffled	0.955	0.956	1.9	24.08	30.59
AP-only	10.164	10.181	2.6	23.38	31.28
Swap	10.152	10.174	2.6	23.41	31.12
Triplet	10.157	10.170	2.7	23.36	31.19

Table 4 : Comparison of different triplet estimation strategies for target-style suppression and content preservation.

Figure 6 : Sensitivity analysis of $\lambda$ . (a) Style and content response retention. (b) Target-style and content scores under different $\lambda$ s.

Figure 7 : Qualitative comparison of Adaptive Suppression Controller variants. The ASC provides more consistent suppression across spatially heterogeneous style regions while retaining the principal content structure.

Metric	$L_{gene}\uparrow$	$C_{style}\uparrow$	$C_{content}\downarrow$	$cs_{style}\downarrow$	$cs_{content}\uparrow$
only_K	0.631	0.200	-0.027	29.52	26.41
only_V	0.649	0.239	-0.013	29.31	26.91
only_Q	0.592	0.177	-0.012	29.73	26.22
w/o Sigmoid	0.591	0.154	0.001	30.40	26.69
Ours	0.664	0.251	-0.043	28.79	27.24

Table 5 : Comparison of different Adaptive Suppression Controller variants. w/o Sigmoid means removing the Sigmoid mapping and replacing it with linear mapping.

Selected patches	$cs_{\mathrm{style}}\downarrow$	$cs_{\mathrm{content}}\uparrow$
Top 20%	28.78	29.76
Random 20%	29.13	29.56
Bottom 20%	29.29	29.34

Table 6 : Effect of patch selection based on style concentration.

Figure 8 : Qualitative comparison of attention intervention pathways. KV achieves target-style suppression visually comparable to QKV without directly intervening on Q.

Pathway	$L_{\mathrm{gene}}\uparrow$	$C_{\mathrm{style}}\uparrow$	$cs_{\mathrm{style}}\downarrow$	$C_{\mathrm{content}}\downarrow$	$cs_{\mathrm{content}}\uparrow$	$C_{\mathrm{DINO}}\uparrow$	SSIM $\uparrow$
Q	0.3709	0.0600	32.42	0.0244	32.95	0.8402	0.6425
K	0.3792	0.0553	32.47	0.0203	32.61	0.8331	0.6164
V	0.4426	0.1059	31.33	0.0565	32.24	0.7408	0.5113
QK	0.4080	0.0759	31.93	0.0347	32.45	0.8336	0.5844
QV	0.5153	0.1473	31.56	0.0893	32.30	0.7145	0.4743
KV	0.5280	0.1581	30.98	0.0885	31.86	0.7194	0.4707

Table 7 : Comparison of attention intervention pathways. Best metric values are shown in bold, and second-best metric values are underlined.

Pathway	$C_{\mathrm{content}}\downarrow$	$cs_{\mathrm{content}}\uparrow$	$C_{\mathrm{DINO}}\uparrow$	SSIM $\uparrow$
Q	0.0620	32.14	0.7477	0.5520
K	0.0616	31.83	0.7388	0.5059
V	0.0560	32.30	0.7492	0.5122
QK	0.0664	30.83	0.6771	0.5021
QV	0.0707	32.26	0.7641	0.5245
KV	0.0520	32.36	0.7890	0.5398

Table 8 : Content preservation results of attention pathways. Best metric values are shown in bold, and second-best metric values are underlined.

Figure 9 : Qualitative cross architecture evaluation on FLUX. CAPE suppresses the characteristic swirling brush strokes and high saturation starry sky textures while preserving the village layout, object structure, and global scene composition.

Method	Offline E2E (s) $\downarrow$	Offline $\Delta$ VRAM (MiB) $\downarrow$	Inference (s/img) $\downarrow$	Inference $\Delta$ VRAM (MiB) $\downarrow$
SD1.4	–	–	2.268	3272
SPEED [ 15 ]	56.647	17116	2.164	3272
ESD-u [ 8 ]	1217.075	15423	2.131	3272
ESD-x [ 8 ]	897.789	10320	2.130	3272
FMN [ 35 ]	262.485	18838	2.145	3272
MACE [ 16 ]	180.005	17844	2.238	3272

Table 9 : Computational efficiency measured for one target concept, one prompt, and one seed.

Figure 10 : Style subspace dimension sensitivity. Small $r$ retains artist specific color and brush traits, large $r$ introduces high frequency artifacts. The mid-range [12,21] consistently achieves clean style removal with high content fidelity.

$k$	Top-k	Bottom-k	Random-k	Random Subspace
1	-0.001592	-0.001611	-0.001830	-0.001770
5	0.008412	0.005367	0.004510	0.004044
10	0.012353	0.012669	0.012005	0.012716
15	0.021918	0.019880	0.018557	0.016450

Table 10 : Target-style suppression with different direction-selection strategies.

Figure 11 : Qualitative stability of CAPE across feature probe positions. We randomly sample 35 positions from the 1000 candidate scheduler indices and independently estimate the style eigenbasis at each position. CAPE maintains consistent target style suppression over a broad range of probe positions, demonstrating that eigenbasis estimation does not rely on a narrowly selected diffusion position.

Figure 12 : Triplet reference sensitivity results. CAPE maintains consistent target style suppression and content structure across different positive content and negative style references, showing stable eigenbasis estimation under different reference choices.

Category	Concept
Artist style	Agnes Martin, Albert Bierstadt, Alphonse Mucha, Amedeo Modigliani, Andre Derain, Aubrey Beardsley, Caspar David Friedrich, Chaim Soutine, Clyfford Still, Dali, David Hockney, Dorothea Tanning, Edgar Degas, Edvard Munch, Egon Schiele, El Greco, El Lissitzky, Ernst Ludwig Kirchner, Fernand Leger, Fra Angelico, Francis Bacon, Francis Picabia, Francisco Goya, Franz Kline, Georges Braque, Gerhard Richter, Giorgio de Chirico, Gustave Courbet, Henri Matisse, Henri Rousseau, Henri de Toulouse-Lautrec, Hieronymus Bosch, Hilma af Klint, Jean Arp, Jean-Francois Millet, Jean-Honore Fragonard, Joan Miro, Johannes Vermeer, John Constable, John Everett Millais, Juan Gris, Kandinsky, Kazimir Malevich, Keith Haring, Leonardo da Vinci, Leonora Carrington, Man Ray, Marcel Duchamp, Mary Cassatt, Max Ernst, Monet, Odilon Redon, Paul Cezanne, Paul Gauguin, Paul Klee, Picasso, Piet Mondrian, Pieter Bruegel the Elder, Rembrandt, Remedios Varo, Rene Magritte, Robert Delaunay, Roy Lichtenstein, Sonia Delaunay, Theo van Doesburg, Thomas Gainsborough, Van Gogh, Warhol, William Blake, William-Adolphe Bouguereau, Winslow Homer, Yayoi Kusama, Yves Tanguy
Object	Mickey Mouse, Naruto, Spider-Man, airplane, ambulance, apple, backpack, banana, bed, bicycle, bird, boat, book, bottle, bowl, bridge, broccoli, bus, butterfly, cake, camera, car, carrot, castle, cat, chair, clock, clothing, comb, corgi, cup, dog, doughnut, drum, eiffel tower, elephant, fire truck, fish, flower, fork, frog, fruit, garbage truck, giraffe, guitar, hamburger, hammer, hat, helmet, horse, ice cream, knife, lamp, laptop, lemon, lesser panda, machine, mango, motorcycle, orange, pear, phone, piano, pineapple, pizza, plate, pyramid, rabbit, rose, sandwich, saxophone, scissors, screwdriver, shark, shoe, snake, sneakers, sofa, spoon, squirrel, statue, strawberry, table, taxi, television, tomato, toothbrush, towel, traffic light, train, tree, truck, trumpet, turtle, umbrella, vegetable, violin, watermelon, wrench, zebra
Celebrity	50 Cent, Aaron Paul, Adam Driver, Adam Sandler, Adele, Al Gore, Al Pacino, Alan Rickman, Alicia Keys, Amy Adams, Angelina Jolie, Anne Hathaway, Ariana Grande, Barack Obama, Ben Affleck, Benedict Cumberbatch, Bernie Sanders, Beyoncé, Bill Clinton, Brad Pitt, Bradley Cooper, Britney Spears, Bruce Willis, Bruno Mars, Bryan Cranston, Cameron Diaz, Celine Dion, Charlize Theron, Chris Evans, Chris Pratt, Christian Bale, Cristiano Ronaldo, David Beckham, David Bowie, Denzel Washington, Donald Trump, Drake, Drew Barrymore, Dua Lipa, Dwayne Johnson, Ed Sheeran, Elon Musk, Elton John, Emma Stone, Emma Watson, George Clooney, Halle Berry, Harrison Ford, Hillary Clinton, Hugh Jackman, Idris Elba, Jack Black, Jack Nicholson, Jackie Chan, Jake Gyllenhaal, Jamie Foxx, Jay Z, Jennifer Aniston, Jennifer Lawrence, Jennifer Lopez, Jim Carrey, Joaquin Phoenix, Joe Biden, Johnny Depp, Julia Roberts, Justin Bieber, Katy Perry, Keanu Reeves, Kevin Hart, Kim Kardashian, Kobe Bryant, Lady Gaga, Leonardo DiCaprio, Lionel Messi, Margot Robbie, Matt Damon, Matthew McConaughey, Miley Cyrus, Morgan Freeman, Natalie Portman, Nicole Kidman, Oprah Winfrey, Reese Witherspoon, Rihanna, Robert Downey Jr, Roger Federer, Sandra Bullock, Scarlett Johansson, Selena Gomez, Serena Williams, Shakira, Sofia Vergara, Stephen Curry, Steve Jobs, Taylor Swift, Vladimir Putin, Whoopi Goldberg, Will Smith, Zendaya, Zoe Saldana

Table S11 : Target concepts used in the distributed causal support analysis.

Category	target concept	Near 8	Middle 8	Far 8
artist	Alphonse Mucha	organic arabesque composition; vintage theatrical poster; symbolist decorative illustration; ornate bookplate design; ornamental art nouveau illustration; flowing botanical linework; decorative belle epoque poster; whiplash curve design	dreamlike religious illustration; spiritual allegorical image; hand-colored visionary print; cosmic poetic scene; symbolic gothic watercolor; visionary romantic illustration; mystical literary watercolor; symbolic biblical drawing	mass-media appropriation art; vivid product illustration; repeated icon composition; retro pop poster; hard-edged popular imagery; pop art screenprint; bold commercial graphic painting; comic-inspired modern artwork
	Amedeo Modigliani	linear figurative painting; restrained expressionist portrait; simplified seated figure; earth-toned studio portrait; modernist nude study; elongated modernist portrait; simplified sculptural face; muted bohemian portrait painting	organic arabesque composition; vintage theatrical poster; symbolist decorative illustration; ornate bookplate design; ornamental art nouveau illustration; flowing botanical linework; decorative belle epoque poster; whiplash curve design	dynamic color-wheel composition; musical cubist color field; radiant modernist geometry; interlocking spectral disks; abstract urban rhythm; vibrating concentric forms; color-based cubist painting; orphic cubist abstraction
	Andre Derain	loose colorful figure painting; radiant decorative landscape; fauvist color painting; vivid non-naturalistic landscape; bold chromatic portrait; high-saturation brushwork; decorative fauvist interior; expressive color-block painting	distorted emotional portrait; agitated brushwork composition; intense psychological landscape; raw chromatic painting; angular urban expressionism; anxious modernist scene; rough impasto portrait; emotionally charged color field	dreamlike religious illustration; spiritual allegorical image; hand-colored visionary print; cosmic poetic scene; symbolic gothic watercolor; visionary romantic illustration; mystical literary watercolor; symbolic biblical drawing
	Caspar David Friedrich	misty mountain painting; luminous pastoral landscape; stormy romantic scene; moonlit wilderness painting; heroic natural panorama; poetic rural landscape; cloud-filled coastal painting; picturesque countryside scene	ornate narrative oil painting; poetic Shakespearean scene; crisp decorative realism; pre-raphaelite painting; jewel-toned medieval portrait; highly detailed romantic figure; botanical literary illustration; luminous Victorian narrative scene	orphic cubist abstraction; simultaneous color circles; rhythmic prismatic painting; luminous geometric color; circular chromatic abstraction; dynamic color-wheel composition; musical cubist color field; radiant modernist geometry
	Edvard Munch	visceral figurative painting; expressionist figure painting; distorted emotional portrait; agitated brushwork composition; intense psychological landscape; raw chromatic painting; angular urban expressionism; anxious modernist scene	vivid non-naturalistic landscape; bold chromatic portrait; high-saturation brushwork; decorative fauvist interior; expressive color-block painting; bright Mediterranean scene; flat intense color composition; wild-beast modern painting	spiritual abstract painting; mystical geometric symbolism; esoteric color diagram; theosophical abstraction; cosmic circular composition; visionary nonobjective painting; symbolic spectral geometry; meditative abstract cosmology
object	airplane	hot air balloon; drone; seaplane; biplane; airship; fighter jet; cargo plane; paraglider	construction site; airport terminal; harbor; subway station; warehouse; service road; public square; city street	laundry basket; mirror; rug; soap dispenser; vacuum cleaner; iron; doormat; pillow

Table S12 : Target concepts and corresponding Near, Middle, and Far non target probes.

Figure S13 : Results over 1000 diffusion steps with an interval of 5 (steps=5,15,20,…,495).

Figure S14 : Results over 1000 diffusion steps with an interval of 5 (steps=500,505,…,995).

Figure S15 : Results under 10 different reference settings.

Prompt	Type	ID	Prompt Description
P1	Anchor	A	A still life of several apples beside a ceramic jug on a wooden table, with a folded cloth and a simple wall behind the objects in the style of Vincent van Gogh.
	Positive	P1	A bright coastal harbor with small sailboats and a stone quay in the style of Vincent van Gogh.
		P2	A cypress-lined country road leading toward distant hills in the style of Vincent van Gogh.
		P3	A wheat field beside a small farmhouse under an open sky in the style of Vincent van Gogh.
		P4	A flowered garden terrace with a table and wicker chairs in the style of Vincent van Gogh.
		P5	A quiet riverside village with a stone bridge and leafy trees in the style of Vincent van Gogh.

Table S13 : Triplet reference prompts used in the sensitivity analysis.

Explore similar work

Oct 1, 2026cs.CV

Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference

Concept erasure removes copyright-protected, privacy-sensitive, or otherwise undesirable concepts from pretrained text-to-image diffusion models to support content governance and compliance. As erasure requests arrive over time, models must remove new targets without undoing prior erasures. Existing methods do not constrain interference across edits: residual perturbations outside the retain set interact and accumulate, degrading unrelated generations and sometimes collapsing previously erased targets into noise. We propose CEASE (Continual Erasure via Adaptive Subspace Editing), a training-free method that imposes two subspace constraints on a closed-form solver. CEASE adds the token representation of the shared replacement to the solver's invariance matrix and, when interference is detected, projects the current update onto the orthogonal complement of dominant output directions extracted from cumulative past updates. A closed-form decomposition attributes the accumulated interference to repeated activation of the shared replacement and overlap between successive update directions, showing that the two constraints suppress these respective sources. Across continual erasure of celebrities, artistic styles, and instances, CEASE achieves the most consistent erase-preserve trade-off, while existing methods either degrade general generation or insufficiently erase targets.

Yongliang Wu, Haori Lu, Jinqi Luo +3

University of Illinois Urbana-Champaign · University of Pennsylvania · National University of Singapore

Concept Erasure Text-To-Image Diffusion Models Diffusion Models Content Moderation Interference Subspace

Ask PDF

Oct 1, 2026cs.CV

RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models

Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation.

Yongliang Wu, Haori Lu, Yulun Wu +3

University of Illinois Urbana-Champaign · University of Pennsylvania · National University of Singapore

Concept Erasure Text-To-Image Diffusion Models Diffusion Models Linear Activation Steering

Ask PDF

May 17, 2026cs.LG

The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation

Generative text-to-image models are typically trained on large-scale web-scraped datasets that include diverse visual content such as copyrighted and stylistically distinctive artworks, raising concerns about ownership, attribution, and the unintended reuse of protected visual expressions. A key issue is that models can learn stylistic patterns from this data and reproduce them in generated outputs without any explicit reference in the prompt. We refer to this phenomenon as The Silent Brush, where such learned styles reappear even when they are not requested. Existing evaluation methods mainly focus on near-duplicate retrieval or membership inference and do not account for this form of unintended stylistic resurfacing across prompts. To address these gaps, we first formulate guiding principles for evaluation of The Silent Brush. We then introduce Art Arena, an evaluation protocol that measures how strongly artworks are encoded, how they interact, and how frequently their stylistic traits reappear in generated outputs without explicit mention in prompts. We evaluate Art Arena on widely used text-to-image diffusion models, including Stable Diffusion v1.5, Stable Diffusion XL (SDXL), and SANA-1.5, and design it to generalize across text-to-image generative systems. Our results show that The Silent Brush arises from differences in representational strength and interaction dynamics between artworks, leading to asymmetric blending in model generations. Code and evaluation resources are available at: https://anonymous.4open.science/r/ArtArena-EBE4.

Ninad Joshi, Ashutosh Ranjan, Vivek Srivastava +1

TCS Research, India

Text-To-Image Diffusion Models Stylistic Fidelity Generative Artificial Intelligence Representational Capacity

Ask PDF