From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

Francesco Gentile; Nicola Dall'Asen; Francesco Tonini; Massimiliano Mancini; Lorenzo Vaquero; Elisa Ricci

cs.CVMar 25, 2026

From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

Authors: Francesco Gentile, Nicola Dall'Asen, Francesco Tonini, Massimiliano Mancini, Lorenzo Vaquero, Elisa Ricci

Organizations: University of Trento · University of Pisa · Fondazione Bruno Kessler

Abstract

As vision-language models are deployed at scale, understanding their internal mechanisms becomes increasingly critical. Existing interpretability methods predominantly rely on activations, making them dataset-dependent, vulnerable to data bias, and often restricted to coarse head-level explanations. We introduce SITH (Semantic Inspection of Transformer Heads), a fully data-free, training-free framework that directly analyzes CLIP's vision transformer in weight space. For each attention head, we decompose its value-output matrix into singular vectors and interpret each one via COMP (Coherent Orthogonal Matching Pursuit), a new algorithm that explains them as sparse, semantically coherent combinations of human-interpretable concepts. We show that SITH yields coherent, faithful intra-head explanations, validated through reconstruction fidelity and interpretability experiments. This allows us to use SITH for precise, interpretable weight-space model edits that amplify or suppress specific concepts, improving downstream performance without retraining. Furthermore, we use SITH to study model adaptation, showing how fine-tuning primarily reweights a stable semantic basis rather than learning entirely new features.

Figures & tables

Figure 1: Comparison between TextSpan [ 23 ] and our proposed SITH applied to CLIP’s vision transformer. TextSpan computes attention-head activations on a large image dataset ( e.g . , ImageNet [ 15 ] ) and aligns them with textual concepts, yielding head-level interpretations ( e.g . , head $h_{1}$ focuses on colors). In contrast, SITH is data-free : it directly decomposes each head’s weights into singular vectors and interprets them via COMP, providing fine-grained, semantically coherent explanations at the vector level ( e.g . , vector $\bm{v}_{2}$ within $h_{1}$ specializes in color green).

	Training-free	Data-free	Explanation Granularity
SAE [ 61 , 52 , 80 ]	✗	✗	Sample-level
TextSpan [ 23 ]	✓	✗	Head-level
SITH (Ours)	✓	✓	Intra-head (singular vector)

Table 1: Comparison between prior works and our SITH .

Figure 2: Overview of SITH and COMP . (left) SITH isolates the Value-Output (VO) matrix of a CLIP ViT attention head and factorizes it via Singular Value Decomposition (SVD), yielding singular vectors $\{\mathbf{v}_{j}\}$ that capture the head’s dominant writing directions. Given a textual concept pool ${\Gamma}$ processed by CLIP’s text encoder $\mathcal{E}_{T}$ , each vector is then explained using COMP, yielding sparse and coherent per-vector interpretations (e.g., “ green ”, “ mint green ”, and “ grass green ”). (right) COMP iteratively decomposes a target singular vector $\bm{\hat{v}}_{j}$ as a sparse, non-negative combination of concept embeddings $\bm{\hat{{\Gamma}}}$ . In Iteration 1 it selects the concept with the highest similarity to the target . The residual vector captures the remaining, unexplained part of the original vector . From Iteration 2 to $K$ , COMP repeatedly chooses additional concepts that are both close to the current residual and semantically consistent with the accumulated concept set (see Eq. 7 ). This process continues until the sparsity budget $K$ is reached.

Figure 3: Interpretability and fidelity scores of SITH evaluated under varying sparsity levels using different decomposition strategies: our proposed COMP, NNOMP [ 55 ] , and top-k.

Layer 23, Head 8, SV 0	Layer 22, Head 7, SV 0
pink red (0.3144) red telephone (0.1571) red and white factors (0.1503) scarlet reds (0.1473) red background (0.1369)	late december (0.1900) winterwear (0.1833) barren trees in winter (0.1729) winter buds (0.1554) winter in valley (0.1383)
Layer 21, Head 11, SV 3	Layer 20, Head 4, SV 0
ocean beach (0.1334) surf culture (0.1170) sandiego (0.1116) catalina island (0.1069) southern california (0.0213)	two girls (0.2121) doublepack (0.2065) two combatants (0.1902) comedy duos (0.1666) two credit cards (0.1612)

Table 2: Examples of singular vectors from the last four layers of ViT-L/14 along with their reconstructed concepts using SITH with COMP (

\lambda=0.3

K=5

). The numbers in parentheses indicate the coefficients assigned to each concept in the reconstruction.

Figure 4: Zero-shot classification accuracy when replacing the original singular vectors of layer

l=23

with their reconstructions using different numbers of concepts and across different datasets [ 15 , 45 , 54 , 35 ] . We also report the accuracy of the original OpenCLIP model, and its accuracy when all heads of the layer are zeroed-out ( i.e . , Zero-ablation).

Figure 5: Top-3 images matched to a singular vector. Each cell shows the top-3 images from CC12M [ 10 ] that are most similar to a specific singular vector, along with the interpretation of that singular vector. See Appendix D of Supp. Mat. for additional examples.

Figure 6: Mean image-concept agreement scores across the last four layers of OpenCLIP [ 32 ] . ⋆ replaces the original concept pool of TextSpan [ 23 ] with ConceptNet 5.5 [ 67 ] .

	Overall	Worst-group
OpenCLIP [ 32 ]	73.5	47.9
w/ Random	72.0	45.1
w/ TextSpan ⋆ [ 23 ]	81.8	68.0
w/ SITH (Ours)	82.7	70.6

Table 3: Zero-shot classification accuracy on Waterbirds [ 62 ] before and after suppressing spurious features. ⋆ indicates recomputed with zero-ablation of heads for zero-shot comparison. “Random” refers to average performance obtained by randomly ablating same number of singular values as our method, averaged over 5 runs.

	Safe Query		Unsafe Query
	T $\rightarrow$ (V $\cup$ V ⋆ )	V $\rightarrow$ (T $\cup$ T ⋆ )	T ⋆ $\rightarrow$ (V $\cup$ V ⋆ )	V ⋆ $\rightarrow$ (T $\cup$ T ⋆ )
Safe-CLIP † [ 57 ]	69.2	73.9	46.3	62.3
OpenCLIP [ 32 ]	75.1	77.0	29.3	38.9
w/ SITH (Ours)	74.5	77.3	29.5	40.4

Table 4: Retrieval R@10 results on the ViSU [ 57 ] test set . Left columns show text-to-image and image-to-text performance when using textual and visual safe queries ( i.e . , T and V , respectively) on a pool of both safe and unsafe data. Right columns use unsafe textual and visual queries ( i.e . , T* and V* , respectively). † Safe-CLIP [ 32 ] is a training-based method specific for NSFW removal that uses OpenAI-CLIP [ 58 ] pretraining.

	Flowers 102 [ 45 ]	FGVC-Aircraft [ 39 ]	DTD [ 12 ]
OpenCLIP [ 32 ]	76.5	36.6	50.1
w/ Random	76.4	36.3	49.9
w/ SITH (Ours)	77.5	36.9	50.9

Table 5: Zero-shot classification accuracy on various datasets before and after editing the model by amplifying/suppressing specific singular values.

Figure 7: Changes in the attention heads after model adaptation. For each attention head ( i.e . square in the plot) of the last four layers of CLIP ViT-L/14, we report the normalized spectral cosine similarity [ 1 ] between the right singular vectors of the pre-trained ( $\bm{W}_{VO}^{pre}$ ) and finetuned ( $\bm{W}_{VO}^{ft}$ ) matrices of that head, across different fine-tuning datasets and adaptation methods. High values correspond to subtle shifts in the head after fine-tuning.

Figure 8: Analysis of

\Delta\bm{W}

with SITH after model adaptation . Using an LLM-as-a-judge we compute the percentage of task singular vectors [ 25 ] aligned with the fine-tuning domain across different benchmarks and adaptation methods.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Concept Pool	Scale	Granularity	Safety Alignment	Language Coverage
TextSpan [ 23 ]	3498	Low	High	English-only
SpLiCE [ 2 ]	15K	Medium	High	English-only
WordNet [ 41 ]	153K	High	Medium	English-only
ConceptNet 5.5 [ 67 ]	1.35M	High	Low	Multilingual

Appendix

Table 6: Comparison of different concept pools along four critical axes: scale ( i.e . , number of concepts), granularity, safety alignment, and language coverage. ConceptNet 5.5 outperforms the alternatives across all dimensions, making it the most suitable choice for our interpretability framework.

Figure 9: Fidelity Score for different Concept Pools. For each reconstruction method, we report the cosine-similarity (averaged across the last four layers of ViT-L-14) between the original singular vectors and their reconstructed versions at different sparsity levels and for different concept pools. We can see that for all methods and sparsity levels, ConceptNet guarantees superior reconstruction capabilities.

Figure 10: Interpretability vs. Fidelity trade-off across Layers 20, 21, and 22 of CLIP ViT-L/14. For each tested layer, we compare the performance of our proposed COMP method against two baselines: Top-

k

selection and NNOMP [ 55 ] . Each point represents a different sparsity level (

K\in\{5,10,20,50\}

). Across all layers, COMP consistently achieves a superior balance between interpretability and fidelity compared to the baselines.

Figure 11: Interpretability vs. Fidelity trade-off for the left singular vectors of the last layer of CLIP ViT-L/14 . We compare the performance of our proposed COMP against two baselines: Top-

k

selection and NNOMP [ 55 ] . Each point represents a different sparsity level (

K\in\{5,10,20,50\}

Left : house rooms	1st Singular Vector		Right : outdoor locations
	• gogglebox (0.1730) • cleaning living room (0.1713) • hotel suite (0.1609) • kitchen bedroom bathroom (0.1470) • room of house (0.1068)	• outdoor concert (0.1937) • park near mall entrance (0.1920) • outside budget (0.1890) • having s outdoors (0.1883) • outdoor stalls (0.1638)
Left : street	2nd Singular Vector		Right : event rooms
	• roadside trees (0.1622) • street price (0.1557) • getting outdoors (0.1546) • product for car owners (0.1370) • street mobile (0.0820)	• locker room (0.2331) • conference center (0.1971) • banquetting (0.1832) • building institution club etc (0.1538) • wedding hall (0.1244)
Left : backyard	3rd Singular Vector		Right : shops
	• balcony bra (0.2164) • accommodation set aside for guests (0.2002) • outdoor home (0.1697) • lawn parties (0.1653) • relaxing on porch (0.1228)	• checkout clerk at supermarket (0.1383) • optical store (0.1296) • store interior (0.1100) • in bookstore (0.1051) • computer store owner (0.0852)
Left : home areas	4th Singular Vector		Right : road, automobile

Appendix

Table 7: Layer 22, Head 2 of ViT-L/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 22, Head 2 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : objects for the upper torso	1st Singular Vector		Right : fabric items
	• pullover sweater (0.1905) • necklacings (0.1726) • museum bust (0.1720) • water on chest (0.1484) • on chest (0.1204)	• frieze pants (0.1977) • pair of short pantlegs (0.1654) • curtains (0.1580) • classic pants (0.1186) • summer trousers (0.1148)
Left : cards	2nd Singular Vector		Right : shirt, table
	• long jacket (0.1569) • invite job applications (0.1460) • card case (0.1157) • invitation card (0.0733) • pc card (0.0571)	• shirt arm (0.3040) • nest table (0.1871) • tablebases (0.1627) • tabletopped (0.1433) • table (0.0630)
Left : head parts	3rd Singular Vector		Right : dress
	• open fence (0.1835) • wig head (0.1754) • hearing (0.1661) • earsies (0.1175) • human ears (0.0388)	• iliotibial band (0.2007) • dress and skirts (0.1926) • shirtdress (0.1881) • drawer under telephone (0.1648) • under dress (0.1175)
Left : dorsal region	4th Singular Vector		Right : body parts

Appendix

Table 8: Layer 22, Head 3 of ViT-L/14 encodes objects / body parts . The first 5 pairs of left/right singular vectors from Layer 22, Head 3 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : clothing	1st Singular Vector		Right : wax et similia
	• interknits (0.1811) • comfortable clothes (0.1682) • embroidered into cloth with sewing (0.1634) • made from cloth (0.1520) • cloth car seat (0.1429)	• leathery skin (0.2421) • encaustics (0.1890) • persulphates (0.1549) • gloss stick (0.1489) • rubber and latex (0.1392)
Left : metal	2nd Singular Vector		Right : leather
	• steel aluminium (0.1945) • played with small metal rod (0.1439) • metal helmet (0.1322) • metal jewelry (0.1264) • piece of metal furniture (0.1173)	• embroidered into cloth with sewing (0.1977) • suede leather (0.1530) • goatskins (0.1333) • fruit leather (0.1299) • soft leather (0.1166)
Left : food	3rd Singular Vector		Right : leather
	• food color (0.1818) • look at paintings of food (0.1671) • drinking yogurt (0.1571) • packaged breakfast food (0.1420) • peanut pastes (0.1356)	• leather flower (0.1400) • leather trades (0.1352) • leatherwork (0.1072) • upholstered with leather (0.1056) • leather case (0.1003)
Left : plastic	4th Singular Vector		Right : steel + drink

Appendix

Table 9: Layer 23, Head 0 of ViT-L/14 encodes materials . The first 5 pairs of left/right singular vectors from Layer 23, Head 0 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Left : women	1st Singular Vector		Right : men
	• group of groups of women (0.2365) • saint agnes eves (0.2199) • women’s writing (0.1835) • girlswear (0.1670) • business girl (0.1587)	• father and son (0.1844) • men’s aesthetic (0.1623) • gay male s (0.1530) • groomsmen (0.1394) • sexy guys (0.0826)
Left : boys	2nd Singular Vector		Right : couples
	• young boy (0.1956) • solo comedian (0.1526) • child in camp (0.1440) • juvenile vagrant (0.1348) • transit worker (0.1348)	• unhappy couples (0.2710) • tag team (0.2025) • co founders (0.1860) • tandem bicycles (0.1748) • couples together (0.0864)
Left : kids	3rd Singular Vector		Right : spouse
	• junior bridesmaids (0.2663) • generic kids (0.1880) • children training (0.1834) • kids who brothers (0.1712) • kids together (0.0932)	• man health worker (0.1322) • clumsy spouse (0.1137) • heterosexual woman in love (0.1052) • testing routine on spouse (0.1015) • spouse scares (0.0781)
Left : two (people)	4th Singular Vector		Right : group

Appendix

Table 10: Layer 23, Head 4 of ViT-L/14 encodes people . The first 5 pairs of left/right singular vectors from Layer 23, Head 4 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : yellowish	1st Singular Vector		Right : red
	• ground zeroes (0.2402) • yellow green color (0.1565) • blue yellow (0.1186) • bluish yellow (0.1040)	• pink red (0.3144) • red telephone (0.1571) • red and white factors (0.1503) • scarlet reds (0.1473) • red background (0.1369)
Left : orange	2nd Singular Vector		Right : purple
	• orange green (0.2026) • cyanodiethylgold (0.1773) • orange blossoms (0.1121) • peach color (0.1075) • orange mint (0.0234)	• royal purple (0.2482) • purple pills (0.1838) • purple thing (0.1546) • purple states (0.1496) • violet purple (0.1063)
Left : yellow	3rd Singular Vector		Right : blue
	• yellowred (0.2565) • yellow press (0.1400) • yellow clothes (0.1292) • yellow fever (0.1128) • yellow hot (0.0585)	• blue pink (0.2813) • blue and white ceramics (0.1795) • prussian blue (0.1376) • blue films (0.1281) • blue hair (0.0649)
Left : light green	4th Singular Vector		Right : orange

Appendix

Table 11: Layer 23, Head 8 of ViT-L/14 encodes colors . The first 5 pairs of left/right singular vectors from Layer 23, Head 8 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : the letter C	1st Singular Vector		Right : the letter M
	• c t splice (0.1914) • cjd (0.1284) • csdl (0.0907) • cpsa (0.0829) • cpsu (0.0688)	• mcos (0.0904) • msfc (0.0805) • mvcs (0.0779) • mcls (0.0678) • µcs (0.0656)
Left : the letter S	2nd Singular Vector		Right : the letter P
	• surname ma (0.1616) • msw (0.1225) • vsv marv (0.0733) • µsv (0.0496)	• p adic numbers (0.1837) • fims (0.1502) • pcas (0.1478) • pbem (0.1368)
Left : the letter D	3rd Singular Vector		Right : the letters ISO
	• dilli bags (0.1720) • ohle (0.1633) • dalk (0.1148) • dillies (0.0817) • dalks (0.0614)	• isopolitical (0.1671) • isonyms (0.1439) • isoniazid (0.1303) • isonym (0.0390) • isonymic (0.0082)
Left : the letter L	4th Singular Vector		Right : the letter S

Appendix

Table 12: Layer 23, Head 11 of ViT-L/14 encodes letters . The first 5 pairs of left/right singular vectors from Layer 23, Head 11 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. In this case, it is interesting to observe that CLIP is able to read the text present in the watermarks of the images, and the explanations correspond to letters present in those watermarks. Unsafe text/images have been redacted/blurred.

Left : shades of white	1st Singular Vector		Right : black
	• wear white dresses (0.1973) • white yellow (0.1605) • beige boxes (0.1477) • cream top (0.1409) • cream colored (0.1324)	• dark kitchens (0.2042) • black top (0.1677) • black on black (0.1609) • people wearing black (0.1290) • black clothes (0.0508)
Left : gold	2nd Singular Vector		Right : gray
	• reddish gold (0.2863) • brown boxes (0.1661) • golden yellow color (0.1634) • double gloucester (0.1504) • peanut butter (0.1290)	• grey white (0.2089) • silver gray hair (0.1531) • gray blue (0.1469) • grey suits (0.1262) • cool grays (0.0999)
Left : brown gray	3rd Singular Vector		Right : white
	• yellowish gray (0.2106) • taupe (0.1897) • brown boxes (0.1656) • bronze component (0.1339) • brown gray (0.0899)	• white black (0.1495) • white shoes (0.1046) • white shirt (0.0934) • white gown (0.0881) • white and black (0.0730)
Left : black	4th Singular Vector		Right : red and blue

Appendix

Table 13: Layer 11, Head 2 of ViT-B/32 encodes colors . The first 5 pairs of left/right singular vectors from Layer 11, Head 2 of ViT-B/32. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Left : men	1st Singular Vector		Right : women
	• man p y (0.1659) • father and son (0.1577) • groomsmen (0.1565) • men’s aesthetic (0.1427) • two men holding hands (0.1390)	• urmila (0.2142) • evil woman (0.2048) • female political activists (0.1991) • scotswomen (0.1749) • women’s article (0.1460)
Left : married couple	2nd Singular Vector		Right : boys
	• married people (0.1279) • relationship through marriage (0.1199) • newlyweds newly married couple (0.1195) • couples together (0.1058) • married couples (0.0999)	• boy monk (0.1720) • making teenaged boys act silly (0.1699) • she male (0.1647) • group boys (0.1644) • topmen (0.1396)
Left : bridesmaids	3rd Singular Vector		Right : men and women
	• junior bridesmaids (0.1675) • pretty dress (0.1286)	• dad and mom (0.2929) • masculine gender (0.1985) • man who indulges women (0.1791) • man in womans clothes (0.1500) • man male and woman (0.1102)
Left : mother (and son)	4th Singular Vector		Right : girls

Appendix

Table 14: Layer 31, Head 7 of ViT-H/14 encodes people . The first 5 pairs of left/right singular vectors from Layer 31, Head 7 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : the letter M	1st Singular Vector		Right : the letter S
	• mln (0.2066) • act reasonable using mind (0.1886) • neuromyoarterial (0.1710) • aaai (0.1417) • haaf net (0.1388)	• csps (0.2065) • and in cases where letter ss is unavailable (0.2058) • say so (0.1826) • sds (0.1625) • stps (0.1597)
Left : the letters A, S	2nd Singular Vector		Right : the letters F, T, P
	• genus alosa (0.1889) • ssaa affiliated (0.1730) • now now ism (0.1612) • nsaim (0.1330) • aas (0.1266)	• ftped (0.1914) • eptfe (0.1771) • people first party (0.1760) • tax etc (0.1623) • pte (0.1446)
Left : the letters T, M	3rd Singular Vector		Right : the letters C, D, E
	• syrt (0.1828) • t y f g (0.1797) • jvm ti (0.1779) • wry mouth (0.1686) • ntim (0.1576)	• d in p aeq (0.2637) • dead ice (0.2117) • eogs (0.2032) • cded (0.1722) • egd (0.0813)
Left : the letter A	4th Singular Vector		Right : the letter C

Appendix

Table 15: Layer 31, Head 11 of ViT-H/14 encodes letters . The first 5 pairs of left/right singular vectors from Layer 31, Head 11 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : festivals	1st Singular Vector		Right : stores
	• megafestivals (0.1719) • bedside book (0.1338) • fix piece of wood furniture (0.1328) • made from wood cotton or linen (0.1222) • handy for mending wood furniture (0.0038)	• museum building (0.2123) • in laundry store (0.1761) • airport lounges (0.1570) • information store p c (0.1409) • retail store (0.1208)
Left : home interior	2nd Singular Vector		Right : public spaces
	• nail wood (0.1616) • front parlor (0.1585) • store things on bookshelf (0.1583) • interior decorating (0.1169) • bedroom storage (0.0656)	• outside hospital (0.1990) • campus festival (0.1683) • courtyard (0.1605) • industrial parks (0.1573) • convention center (0.1518)
Left : training class	3rd Singular Vector		Right : fairs
	• learning about cartooning (0.1569) • corporate in service training (0.1345) • training class (0.0892) • employee training (0.0141) • teacher training (0.0055)	• make bathroom walls (0.2191) • at fair (0.1881) • bus depot booth (0.1594) • comiket (0.1589) • art fair (0.1474)
Left : hall	4th Singular Vector		Right : outdoor

Appendix

Table 16: Layer 31, Head 12 of ViT-H/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 31, Head 12 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

. Unsafe text/images have been redacted/blurred.

Left : green, brown	1st Singular Vector		Right : indigo purple
	• deadspace (0.1830) • mint green (0.1571) • brown hooded parrot (0.1319) • brown teal (0.1070) • mint chocolate chip (0.1044)	• navy thing (0.2547) • facebooks (0.2191) • sodalite (0.1731) • indigo paper (0.1728) • navy look (0.1383)
Left : red and blue	2nd Singular Vector		Right : black purple
	• cyanotype (0.1935) • red shouldered macaw (0.1549) • korean air (0.1544) • teal (0.1537) • turquoise thing (0.0407)	• black purple (0.2681) • purple black (0.2144) • amethysts (0.1607) • asexual (0.1566) • purples (0.0416)
Left : red, black, blue	3rd Singular Vector		Right : green, purple
	• red and black (0.3333) • blue coal (0.1386) • blue blacks (0.1367) • blue red (0.1010) • black and blue (0.0736)	• lakers and celtics (0.1884) • rhododendron viscosum (0.1854) • green purple (0.1783) • purplish green (0.1078)
Left : green red	4th Singular Vector		Right : light blue

Appendix

Table 17: Layer 31, Head 13 of ViT-H/14 encodes colors . The first 5 pairs of left/right singular vectors from Layer 31, Head 13 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Left : 4	1st Singular Vector		Right : 8
	• 45 (0.3989) • four lags (0.1110) • four books (0.1019) • four stroking (0.0958) • four flushes (0.0770)	• nine parts (0.1750) • 78 (0.1652) • group of eight (0.1271) • 8º (0.0924)
Left : 6	2nd Singular Vector		Right : 4, 3
	• 56 (0.2728) • six ksitigarbhas (0.1804) • six copies (0.1371) • six footedness (0.1312) • five six (0.0541)	• rounded to 3.14 (0.1518) • 14er (0.1353) • 3 4 (0.1233) • fourteen (0.0996)
Left : 4, 6, 8	3rd Singular Vector		Right : 5, 3, 1
	• 6in4 (0.1865) • 84000 (0.1207) • 4 8 0 (0.0710) • 4 8 6 (0.0670) • 486 (0.0646)	• 51 percent (0.1256) • 357 (0.1095) • 3 1 1 (0.0960) • 5 1 (0.0579) • 51 (0.0515)
Left : 7, 9	4th Singular Vector		Right : 12, 16

Appendix

Table 18: Layer 22, Head 0 of MobileCLIP-L/14 encodes numbers . The first 5 pairs of left/right singular vectors from Layer 22, Head 0 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Left : home	1st Singular Vector		Right : public facilities
	• living room plus kitchen (0.1695) • home straights (0.1165) • most homes (0.1011) • used in homes (0.0992) • home dwellers (0.0873)	• restaurant hotel (0.1550) • wash clothes at laundromat (0.1452) • classrooms offices (0.1448) • at restaurant cafeteria coffee shop etc (0.0633) • work in cafeteria (0.0420)
Left : room	2nd Singular Vector		Right : outdoor shop
	• accomodating (0.1775) • room connector (0.1659) • found in hotel room (0.1347) • room hotel (0.1264) • employed to decorate hotel rooms (0.1171)	• outdoor shop (0.2320) • pedestrian mall (0.1341) • street markets (0.1314)
Left : neighborhood	3rd Singular Vector		Right : exhibit hall
	• often apartments over restaurants (0.2194) • in neighbourhood of (0.0976) • in close neighborhood (0.0686) • in neighborhood (0.0277)	• trade fair (0.1726) • go to exhibit hall (0.1439) • gigafactory (0.1425) • international exposition (0.1045) • fairgrounds (0.0935)
Left : bedroom	4th Singular Vector		Right : outside eating

Appendix

Table 19: Layer 22, Head 10 of MobileCLIP-L/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 22, Head 10 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Left : leather	1st Singular Vector		Right : cotton
	• leather and (0.1932) • long leather (0.1653) • leatherers (0.1418) • soft leather (0.0757) • real leather (0.0496)	• polycotton (0.1104) • amigurumi (0.0997) • canvaswork (0.0862) • machine knit (0.0844)
Left : silicone	2nd Singular Vector		Right : metal
	• siliconise (0.1965) • rubber toy (0.1474) • clay based (0.1371) • silicone rubbers (0.0746)	• steel tin (0.1858) • metallometallations (0.1193) • typically made of steel and (0.0944) • lightest metal (0.0900) • converted into steel (0.0781)
Left : plastic	3rd Singular Vector		Right : ceramic
	• matte (0.1836) • plastic box (0.1596) • plastic rubber (0.1364) • plastic plods (0.1216)	• glazed tile (0.1506) • stoneware (0.1014) • ceramometals (0.0879) • ceramic glaze (0.0647) • faience (0.0559)
Left : silk	4th Singular Vector		Right : wood

Appendix

Table 20: Layer 23, Head 13 of MobileCLIP-L/14 encodes materials . The first 5 pairs of left/right singular vectors from Layer 23, Head 13 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with

\lambda=0.3

and

K=5

Panel of Table 21 — Table 21: The prompt used to evaluate the monosemanticity of concept sets extracted by different methods ( Sec. 4.1 )

Panel of Table 22 — Table 22: The prompt used to evaluate the alignment between retrieved images and textual interpretations ( Sec. 4.2 )

Panel of Table 23 — Table 23: The prompt used for the detection of spurious concepts ( Sec. 5.1 )

Panel of Table 24 — Table 24: The prompt used for removing NSFW concepts ( Sec. 5.2 )

Panel of Table 25 — Table 25: The prompt used to evaluate the alignment of task singular vectors to the Flowers102 domain ( Sec. 6 )

Panel of Table 26 — Table 26: The prompt used to evaluate the alignment of task singular vectors to the Oxford-IIIT Pet domain ( Sec. 6 )

Panel of Table 27 — Table 27: The prompt used to evaluate the alignment of task singular vectors to the CUB-200 domain ( Sec. 6 )

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

Explaining CLIP Zero-shot Predictions Through Concepts
Mar 30, 2026Onat Ozdemir, Anders Christensen, Stephan Alaniz +2VLM Interpretability Explainable Image Classification
School of Informatics, University of Edinburgh · Dept. of Computer Eng., Middle East Technical University (METU) · Orbital +8
Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision
Apr 14, 2026Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4Transformer Interpretability Vision Transformer
Rutgers University · University of California San Diego
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
May 15, 2026Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1Distribution Shift Robustness VLM Robustness
University of Tübingen ∗ · University of Tübingen

Mar 30, 2026J/K move · Enter open · S save

Explaining CLIP Zero-shot Predictions Through Concepts

Onat Ozdemir, Anders Christensen, Stephan Alaniz, Zeynep Akata+1

School of Informatics, University of Edinburgh · Dept. of Computer Eng., Middle East Technical University (METU) · Orbital+8

VLM Interpretability Explainable Image Classification Concept-Based Explanations Zero-Shot Image Classification

Open paper AskPDF arXiv

Large-scale vision-language models such as CLIP have achieved remarkable success in zero-shot image recognition, yet their predictions remain largely opaque to human understanding. In contrast, Concept Bottleneck Models provide interpretable intermediate representations by reasoning through human-defined concepts, but they rely on concept supervision and lack the ability to generalize to unseen classes. We introduce EZPC that bridges these two paradigms by explaining CLIP's zero-shot predictions through human-understandable concepts. Our method projects CLIP's joint image-text embeddings into a concept space learned from language descriptions, enabling faithful and transparent explanations without additional supervision. The model learns this projection via a combination of alignment and reconstruction objectives, ensuring that concept activations preserve CLIP's semantic structure while remaining interpretable. Extensive experiments on five benchmark datasets, CIFAR-100, CUB-200-2011, Places365, ImageNet-100, and ImageNet-1k, demonstrate that our approach maintains CLIP's strong zero-shot classification accuracy while providing meaningful concept-level explanations. By grounding open-vocabulary predictions in explicit semantic concepts, our method offers a principled step toward interpretable and trustworthy vision-language models. Code is available at https://github.com/oonat/ezpc.

Abstract

Transformer Interpretability

Explore similar work

Explaining CLIP Zero-shot Predictions Through Concepts

Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models