As vision-language models are deployed at scale, understanding their internal mechanisms becomes increasingly critical. Existing interpretability methods predominantly rely on activations, making them dataset-dependent, vulnerable to data bias, and often restricted to coarse head-level explanations. We introduce SITH (Semantic Inspection of Transformer Heads), a fully data-free, training-free framework that directly analyzes CLIP's vision transformer in weight space. For each attention head, we decompose its value-output matrix into singular vectors and interpret each one via COMP (Coherent Orthogonal Matching Pursuit), a new algorithm that explains them as sparse, semantically coherent combinations of human-interpretable concepts. We show that SITH yields coherent, faithful intra-head explanations, validated through reconstruction fidelity and interpretability experiments. This allows us to use SITH for precise, interpretable weight-space model edits that amplify or suppress specific concepts, improving downstream performance without retraining. Furthermore, we use SITH to study model adaptation, showing how fine-tuning primarily reweights a stable semantic basis rather than learning entirely new features.
Figures & tables
Figure 1: Comparison between TextSpan [ 23 ] and our proposed SITH applied to CLIP’s vision transformer. TextSpan computes attention-head activations on a large image dataset ( e.g . , ImageNet [ 15 ] ) and aligns them with textual concepts, yielding head-level interpretations ( e.g . , head h1 focuses on colors). In contrast, SITH is data-free : it directly decomposes each head’s weights into singular vectors and interprets them via COMP, providing fine-grained, semantically coherent explanations at the vector level ( e.g . , vector v2 within h1 specializes in color green).
Training-free
Data-free
Explanation Granularity
SAE [ 61 , 52 , 80 ]
✗
✗
Sample-level
TextSpan [ 23 ]
✓
✗
Head-level
SITH (Ours)
✓
✓
Intra-head (singular vector)
Table 1: Comparison between prior works and our SITH .
Figure 2: Overview of SITH and COMP . (left) SITH isolates the Value-Output (VO) matrix of a CLIP ViT attention head and factorizes it via Singular Value Decomposition (SVD), yielding singular vectors {vj} that capture the head’s dominant writing directions. Given a textual concept pool Γ processed by CLIP’s text encoder ET , each vector is then explained using COMP, yielding sparse and coherent per-vector interpretations (e.g., “ green ”, “ mint green ”, and “ grass green ”). (right) COMP iteratively decomposes a target singular vector v^j as a sparse, non-negative combination of concept embeddings Γ^ . In Iteration 1 it selects the concept with the highest similarity to the target . The residual vector captures the remaining, unexplained part of the original vector . From Iteration 2 to K , COMP repeatedly chooses additional concepts that are both close to the current residual and semantically consistent with the accumulated concept set (see Eq. 7 ). This process continues until the sparsity budget K is reached.
Figure 3: Interpretability and fidelity scores of SITH evaluated under varying sparsity levels using different decomposition strategies: our proposed COMP, NNOMP [ 55 ] , and top-k.
Layer 23, Head 8, SV 0
Layer 22, Head 7, SV 0
pink red (0.3144) red telephone (0.1571) red and white factors (0.1503) scarlet reds (0.1473) red background (0.1369)
late december (0.1900) winterwear (0.1833) barren trees in winter (0.1729) winter buds (0.1554) winter in valley (0.1383)
Layer 21, Head 11, SV 3
Layer 20, Head 4, SV 0
ocean beach (0.1334) surf culture (0.1170) sandiego (0.1116) catalina island (0.1069) southern california (0.0213)
two girls (0.2121) doublepack (0.2065) two combatants (0.1902) comedy duos (0.1666) two credit cards (0.1612)
Table 2: Examples of singular vectors from the last four layers of ViT-L/14 along with their reconstructed concepts using SITH with COMP ( λ=0.3 , K=5 ). The numbers in parentheses indicate the coefficients assigned to each concept in the reconstruction.
Figure 4: Zero-shot classification accuracy when replacing the original singular vectors of layer l=23 with their reconstructions using different numbers of concepts and across different datasets [ 15 , 45 , 54 , 35 ] . We also report the accuracy of the original OpenCLIP model, and its accuracy when all heads of the layer are zeroed-out ( i.e . , Zero-ablation).
Figure 5: Top-3 images matched to a singular vector. Each cell shows the top-3 images from CC12M [ 10 ] that are most similar to a specific singular vector, along with the interpretation of that singular vector. See Appendix D of Supp. Mat. for additional examples.
Figure 6: Mean image-concept agreement scores across the last four layers of OpenCLIP [ 32 ] . ⋆ replaces the original concept pool of TextSpan [ 23 ] with ConceptNet 5.5 [ 67 ] .
Overall
Worst-group
OpenCLIP [ 32 ]
73.5
47.9
w/ Random
72.0
45.1
w/ TextSpan ⋆ [ 23 ]
81.8
68.0
w/ SITH (Ours)
82.7
70.6
Table 3: Zero-shot classification accuracy on Waterbirds [ 62 ] before and after suppressing spurious features. ⋆ indicates recomputed with zero-ablation of heads for zero-shot comparison. “Random” refers to average performance obtained by randomly ablating same number of singular values as our method, averaged over 5 runs.
Safe Query
Unsafe Query
T → (V ∪ V ⋆ )
V → (T ∪ T ⋆ )
T ⋆ → (V ∪ V ⋆ )
V ⋆ → (T ∪ T ⋆ )
Safe-CLIP † [ 57 ]
69.2
73.9
46.3
62.3
OpenCLIP [ 32 ]
75.1
77.0
29.3
38.9
w/ SITH (Ours)
74.5
77.3
29.5
40.4
Table 4: Retrieval R@10 results on the ViSU [ 57 ] test set . Left columns show text-to-image and image-to-text performance when using textual and visual safe queries ( i.e . , T and V , respectively) on a pool of both safe and unsafe data. Right columns use unsafe textual and visual queries ( i.e . , T* and V* , respectively). † Safe-CLIP [ 32 ] is a training-based method specific for NSFW removal that uses OpenAI-CLIP [ 58 ] pretraining.
Flowers 102 [ 45 ]
FGVC-Aircraft [ 39 ]
DTD [ 12 ]
OpenCLIP [ 32 ]
76.5
36.6
50.1
w/ Random
76.4
36.3
49.9
w/ SITH (Ours)
77.5
36.9
50.9
Table 5: Zero-shot classification accuracy on various datasets before and after editing the model by amplifying/suppressing specific singular values.
Figure 7: Changes in the attention heads after model adaptation. For each attention head ( i.e . square in the plot) of the last four layers of CLIP ViT-L/14, we report the normalized spectral cosine similarity [ 1 ] between the right singular vectors of the pre-trained ( WVOpre ) and finetuned ( WVOft ) matrices of that head, across different fine-tuning datasets and adaptation methods. High values correspond to subtle shifts in the head after fine-tuning.
Figure 8: Analysis of ΔW with SITH after model adaptation . Using an LLM-as-a-judge we compute the percentage of task singular vectors [ 25 ] aligned with the fine-tuning domain across different benchmarks and adaptation methods.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Concept Pool
Scale
Granularity
Safety Alignment
Language Coverage
TextSpan [ 23 ]
3498
Low
High
English-only
SpLiCE [ 2 ]
15K
Medium
High
English-only
WordNet [ 41 ]
153K
High
Medium
English-only
ConceptNet 5.5 [ 67 ]
1.35M
High
Low
Multilingual
Appendix
Table 6: Comparison of different concept pools along four critical axes: scale ( i.e . , number of concepts), granularity, safety alignment, and language coverage. ConceptNet 5.5 outperforms the alternatives across all dimensions, making it the most suitable choice for our interpretability framework.
Figure 9: Fidelity Score for different Concept Pools. For each reconstruction method, we report the cosine-similarity (averaged across the last four layers of ViT-L-14) between the original singular vectors and their reconstructed versions at different sparsity levels and for different concept pools. We can see that for all methods and sparsity levels, ConceptNet guarantees superior reconstruction capabilities.
Figure 10: Interpretability vs. Fidelity trade-off across Layers 20, 21, and 22 of CLIP ViT-L/14. For each tested layer, we compare the performance of our proposed COMP method against two baselines: Top- k selection and NNOMP [ 55 ] . Each point represents a different sparsity level ( K∈{5,10,20,50} ). Across all layers, COMP consistently achieves a superior balance between interpretability and fidelity compared to the baselines.
Figure 11: Interpretability vs. Fidelity trade-off for the left singular vectors of the last layer of CLIP ViT-L/14 . We compare the performance of our proposed COMP against two baselines: Top- k selection and NNOMP [ 55 ] . Each point represents a different sparsity level ( K∈{5,10,20,50} ).
Left : house rooms
1st Singular Vector
Right : outdoor locations
• gogglebox (0.1730) • cleaning living room (0.1713) • hotel suite (0.1609) • kitchen bedroom bathroom (0.1470) • room of house (0.1068)
• outdoor concert (0.1937) • park near mall entrance (0.1920) • outside budget (0.1890) • having s outdoors (0.1883) • outdoor stalls (0.1638)
Left : street
2nd Singular Vector
Right : event rooms
• roadside trees (0.1622) • street price (0.1557) • getting outdoors (0.1546) • product for car owners (0.1370) • street mobile (0.0820)
• locker room (0.2331) • conference center (0.1971) • banquetting (0.1832) • building institution club etc (0.1538) • wedding hall (0.1244)
Left : backyard
3rd Singular Vector
Right : shops
• balcony bra (0.2164) • accommodation set aside for guests (0.2002) • outdoor home (0.1697) • lawn parties (0.1653) • relaxing on porch (0.1228)
• checkout clerk at supermarket (0.1383) • optical store (0.1296) • store interior (0.1100) • in bookstore (0.1051) • computer store owner (0.0852)
Left : home areas
4th Singular Vector
Right : road, automobile
Appendix
Table 7: Layer 22, Head 2 of ViT-L/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 22, Head 2 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : objects for the upper torso
1st Singular Vector
Right : fabric items
• pullover sweater (0.1905) • necklacings (0.1726) • museum bust (0.1720) • water on chest (0.1484) • on chest (0.1204)
• open fence (0.1835) • wig head (0.1754) • hearing (0.1661) • earsies (0.1175) • human ears (0.0388)
• iliotibial band (0.2007) • dress and skirts (0.1926) • shirtdress (0.1881) • drawer under telephone (0.1648) • under dress (0.1175)
Left : dorsal region
4th Singular Vector
Right : body parts
Appendix
Table 8: Layer 22, Head 3 of ViT-L/14 encodes objects / body parts . The first 5 pairs of left/right singular vectors from Layer 22, Head 3 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : clothing
1st Singular Vector
Right : wax et similia
• interknits (0.1811) • comfortable clothes (0.1682) • embroidered into cloth with sewing (0.1634) • made from cloth (0.1520) • cloth car seat (0.1429)
• steel aluminium (0.1945) • played with small metal rod (0.1439) • metal helmet (0.1322) • metal jewelry (0.1264) • piece of metal furniture (0.1173)
• embroidered into cloth with sewing (0.1977) • suede leather (0.1530) • goatskins (0.1333) • fruit leather (0.1299) • soft leather (0.1166)
Left : food
3rd Singular Vector
Right : leather
• food color (0.1818) • look at paintings of food (0.1671) • drinking yogurt (0.1571) • packaged breakfast food (0.1420) • peanut pastes (0.1356)
• leather flower (0.1400) • leather trades (0.1352) • leatherwork (0.1072) • upholstered with leather (0.1056) • leather case (0.1003)
Left : plastic
4th Singular Vector
Right : steel + drink
Appendix
Table 9: Layer 23, Head 0 of ViT-L/14 encodes materials . The first 5 pairs of left/right singular vectors from Layer 23, Head 0 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Left : women
1st Singular Vector
Right : men
• group of groups of women (0.2365) • saint agnes eves (0.2199) • women’s writing (0.1835) • girlswear (0.1670) • business girl (0.1587)
• father and son (0.1844) • men’s aesthetic (0.1623) • gay male s (0.1530) • groomsmen (0.1394) • sexy guys (0.0826)
Left : boys
2nd Singular Vector
Right : couples
• young boy (0.1956) • solo comedian (0.1526) • child in camp (0.1440) • juvenile vagrant (0.1348) • transit worker (0.1348)
• unhappy couples (0.2710) • tag team (0.2025) • co founders (0.1860) • tandem bicycles (0.1748) • couples together (0.0864)
Left : kids
3rd Singular Vector
Right : spouse
• junior bridesmaids (0.2663) • generic kids (0.1880) • children training (0.1834) • kids who brothers (0.1712) • kids together (0.0932)
• man health worker (0.1322) • clumsy spouse (0.1137) • heterosexual woman in love (0.1052) • testing routine on spouse (0.1015) • spouse scares (0.0781)
Left : two (people)
4th Singular Vector
Right : group
Appendix
Table 10: Layer 23, Head 4 of ViT-L/14 encodes people . The first 5 pairs of left/right singular vectors from Layer 23, Head 4 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : yellowish
1st Singular Vector
Right : red
• ground zeroes (0.2402) • yellow green color (0.1565) • blue yellow (0.1186) • bluish yellow (0.1040)
• pink red (0.3144) • red telephone (0.1571) • red and white factors (0.1503) • scarlet reds (0.1473) • red background (0.1369)
Left : orange
2nd Singular Vector
Right : purple
• orange green (0.2026) • cyanodiethylgold (0.1773) • orange blossoms (0.1121) • peach color (0.1075) • orange mint (0.0234)
• royal purple (0.2482) • purple pills (0.1838) • purple thing (0.1546) • purple states (0.1496) • violet purple (0.1063)
• blue pink (0.2813) • blue and white ceramics (0.1795) • prussian blue (0.1376) • blue films (0.1281) • blue hair (0.0649)
Left : light green
4th Singular Vector
Right : orange
Appendix
Table 11: Layer 23, Head 8 of ViT-L/14 encodes colors . The first 5 pairs of left/right singular vectors from Layer 23, Head 8 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : the letter C
1st Singular Vector
Right : the letter M
• c t splice (0.1914) • cjd (0.1284) • csdl (0.0907) • cpsa (0.0829) • cpsu (0.0688)
Table 12: Layer 23, Head 11 of ViT-L/14 encodes letters . The first 5 pairs of left/right singular vectors from Layer 23, Head 11 of ViT-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . In this case, it is interesting to observe that CLIP is able to read the text present in the watermarks of the images, and the explanations correspond to letters present in those watermarks. Unsafe text/images have been redacted/blurred.
Left : shades of white
1st Singular Vector
Right : black
• wear white dresses (0.1973) • white yellow (0.1605) • beige boxes (0.1477) • cream top (0.1409) • cream colored (0.1324)
• dark kitchens (0.2042) • black top (0.1677) • black on black (0.1609) • people wearing black (0.1290) • black clothes (0.0508)
Left : gold
2nd Singular Vector
Right : gray
• reddish gold (0.2863) • brown boxes (0.1661) • golden yellow color (0.1634) • double gloucester (0.1504) • peanut butter (0.1290)
• yellowish gray (0.2106) • taupe (0.1897) • brown boxes (0.1656) • bronze component (0.1339) • brown gray (0.0899)
• white black (0.1495) • white shoes (0.1046) • white shirt (0.0934) • white gown (0.0881) • white and black (0.0730)
Left : black
4th Singular Vector
Right : red and blue
Appendix
Table 13: Layer 11, Head 2 of ViT-B/32 encodes colors . The first 5 pairs of left/right singular vectors from Layer 11, Head 2 of ViT-B/32. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Left : men
1st Singular Vector
Right : women
• man p y (0.1659) • father and son (0.1577) • groomsmen (0.1565) • men’s aesthetic (0.1427) • two men holding hands (0.1390)
• married people (0.1279) • relationship through marriage (0.1199) • newlyweds newly married couple (0.1195) • couples together (0.1058) • married couples (0.0999)
• boy monk (0.1720) • making teenaged boys act silly (0.1699) • she male (0.1647) • group boys (0.1644) • topmen (0.1396)
• dad and mom (0.2929) • masculine gender (0.1985) • man who indulges women (0.1791) • man in womans clothes (0.1500) • man male and woman (0.1102)
Left : mother (and son)
4th Singular Vector
Right : girls
Appendix
Table 14: Layer 31, Head 7 of ViT-H/14 encodes people . The first 5 pairs of left/right singular vectors from Layer 31, Head 7 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : the letter M
1st Singular Vector
Right : the letter S
• mln (0.2066) • act reasonable using mind (0.1886) • neuromyoarterial (0.1710) • aaai (0.1417) • haaf net (0.1388)
• csps (0.2065) • and in cases where letter ss is unavailable (0.2058) • say so (0.1826) • sds (0.1625) • stps (0.1597)
Left : the letters A, S
2nd Singular Vector
Right : the letters F, T, P
• genus alosa (0.1889) • ssaa affiliated (0.1730) • now now ism (0.1612) • nsaim (0.1330) • aas (0.1266)
• ftped (0.1914) • eptfe (0.1771) • people first party (0.1760) • tax etc (0.1623) • pte (0.1446)
Left : the letters T, M
3rd Singular Vector
Right : the letters C, D, E
• syrt (0.1828) • t y f g (0.1797) • jvm ti (0.1779) • wry mouth (0.1686) • ntim (0.1576)
• d in p aeq (0.2637) • dead ice (0.2117) • eogs (0.2032) • cded (0.1722) • egd (0.0813)
Left : the letter A
4th Singular Vector
Right : the letter C
Appendix
Table 15: Layer 31, Head 11 of ViT-H/14 encodes letters . The first 5 pairs of left/right singular vectors from Layer 31, Head 11 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : festivals
1st Singular Vector
Right : stores
• megafestivals (0.1719) • bedside book (0.1338) • fix piece of wood furniture (0.1328) • made from wood cotton or linen (0.1222) • handy for mending wood furniture (0.0038)
• museum building (0.2123) • in laundry store (0.1761) • airport lounges (0.1570) • information store p c (0.1409) • retail store (0.1208)
Left : home interior
2nd Singular Vector
Right : public spaces
• nail wood (0.1616) • front parlor (0.1585) • store things on bookshelf (0.1583) • interior decorating (0.1169) • bedroom storage (0.0656)
• outside hospital (0.1990) • campus festival (0.1683) • courtyard (0.1605) • industrial parks (0.1573) • convention center (0.1518)
Left : training class
3rd Singular Vector
Right : fairs
• learning about cartooning (0.1569) • corporate in service training (0.1345) • training class (0.0892) • employee training (0.0141) • teacher training (0.0055)
• make bathroom walls (0.2191) • at fair (0.1881) • bus depot booth (0.1594) • comiket (0.1589) • art fair (0.1474)
Left : hall
4th Singular Vector
Right : outdoor
Appendix
Table 16: Layer 31, Head 12 of ViT-H/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 31, Head 12 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 . Unsafe text/images have been redacted/blurred.
Left : green, brown
1st Singular Vector
Right : indigo purple
• deadspace (0.1830) • mint green (0.1571) • brown hooded parrot (0.1319) • brown teal (0.1070) • mint chocolate chip (0.1044)
• navy thing (0.2547) • facebooks (0.2191) • sodalite (0.1731) • indigo paper (0.1728) • navy look (0.1383)
Left : red and blue
2nd Singular Vector
Right : black purple
• cyanotype (0.1935) • red shouldered macaw (0.1549) • korean air (0.1544) • teal (0.1537) • turquoise thing (0.0407)
• black purple (0.2681) • purple black (0.2144) • amethysts (0.1607) • asexual (0.1566) • purples (0.0416)
Left : red, black, blue
3rd Singular Vector
Right : green, purple
• red and black (0.3333) • blue coal (0.1386) • blue blacks (0.1367) • blue red (0.1010) • black and blue (0.0736)
• lakers and celtics (0.1884) • rhododendron viscosum (0.1854) • green purple (0.1783) • purplish green (0.1078)
Left : green red
4th Singular Vector
Right : light blue
Appendix
Table 17: Layer 31, Head 13 of ViT-H/14 encodes colors . The first 5 pairs of left/right singular vectors from Layer 31, Head 13 of ViT-H/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Left : 4
1st Singular Vector
Right : 8
• 45 (0.3989) • four lags (0.1110) • four books (0.1019) • four stroking (0.0958) • four flushes (0.0770)
• nine parts (0.1750) • 78 (0.1652) • group of eight (0.1271) • 8º (0.0924)
Left : 6
2nd Singular Vector
Right : 4, 3
• 56 (0.2728) • six ksitigarbhas (0.1804) • six copies (0.1371) • six footedness (0.1312) • five six (0.0541)
Table 18: Layer 22, Head 0 of MobileCLIP-L/14 encodes numbers . The first 5 pairs of left/right singular vectors from Layer 22, Head 0 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Left : home
1st Singular Vector
Right : public facilities
• living room plus kitchen (0.1695) • home straights (0.1165) • most homes (0.1011) • used in homes (0.0992) • home dwellers (0.0873)
• restaurant hotel (0.1550) • wash clothes at laundromat (0.1452) • classrooms offices (0.1448) • at restaurant cafeteria coffee shop etc (0.0633) • work in cafeteria (0.0420)
Left : room
2nd Singular Vector
Right : outdoor shop
• accomodating (0.1775) • room connector (0.1659) • found in hotel room (0.1347) • room hotel (0.1264) • employed to decorate hotel rooms (0.1171)
• often apartments over restaurants (0.2194) • in neighbourhood of (0.0976) • in close neighborhood (0.0686) • in neighborhood (0.0277)
• trade fair (0.1726) • go to exhibit hall (0.1439) • gigafactory (0.1425) • international exposition (0.1045) • fairgrounds (0.0935)
Left : bedroom
4th Singular Vector
Right : outside eating
Appendix
Table 19: Layer 22, Head 10 of MobileCLIP-L/14 encodes locations . The first 5 pairs of left/right singular vectors from Layer 22, Head 10 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Left : leather
1st Singular Vector
Right : cotton
• leather and (0.1932) • long leather (0.1653) • leatherers (0.1418) • soft leather (0.0757) • real leather (0.0496)
Table 20: Layer 23, Head 13 of MobileCLIP-L/14 encodes materials . The first 5 pairs of left/right singular vectors from Layer 23, Head 13 of MobileCLIP-L/14. For each singular vector, we display the top-4 images from CC12M [ 10 ] most similar to it, along with the explanation generated by COMP with λ=0.3 and K=5 .
Table 21: The prompt used to evaluate the monosemanticity of concept sets extracted by different methods ( Sec. 4.1 )
Table 22: The prompt used to evaluate the alignment between retrieved images and textual interpretations ( Sec. 4.2 )
Table 23: The prompt used for the detection of spurious concepts ( Sec. 5.1 )
Table 24: The prompt used for removing NSFW concepts ( Sec. 5.2 )
Table 25: The prompt used to evaluate the alignment of task singular vectors to the Flowers102 domain ( Sec. 6 )
Table 26: The prompt used to evaluate the alignment of task singular vectors to the Oxford-IIIT Pet domain ( Sec. 6 )
Table 27: The prompt used to evaluate the alignment of task singular vectors to the CUB-200 domain ( Sec. 6 )
Large-scale vision-language models such as CLIP have achieved remarkable success in zero-shot image recognition, yet their predictions remain largely opaque to human understanding. In contrast, Concept Bottleneck Models provide interpretable intermediate representations by reasoning through human-defined concepts, but they rely on concept supervision and lack the ability to generalize to unseen classes. We introduce EZPC that bridges these two paradigms by explaining CLIP's zero-shot predictions through human-understandable concepts. Our method projects CLIP's joint image-text embeddings into a concept space learned from language descriptions, enabling faithful and transparent explanations without additional supervision. The model learns this projection via a combination of alignment and reconstruction objectives, ensuring that concept activations preserve CLIP's semantic structure while remaining interpretable. Extensive experiments on five benchmark datasets, CIFAR-100, CUB-200-2011, Places365, ImageNet-100, and ImageNet-1k, demonstrate that our approach maintains CLIP's strong zero-shot classification accuracy while providing meaningful concept-level explanations. By grounding open-vocabulary predictions in explicit semantic concepts, our method offers a principled step toward interpretable and trustworthy vision-language models. Code is available at https://github.com/oonat/ezpc.
Onat Ozdemir, Anders Christensen, Stephan Alaniz +2
School of Informatics, University of Edinburgh · Dept. of Computer Eng., Middle East Technical University (METU) · Orbital +8
Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.
Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4
Rutgers University · University of California San Diego
Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks. Code is publicly available at: https://github.com/Fabian-Mor/sae-ft.