ISBC: Interpretable Sparse Binary Coding of Pretrained Text Embeddings
Organizations: Joint Research Centre European Commission Seville (SP) · Department of Global Political Studies Malmö University Malmö (SE)
Abstract
Pretrained text embeddings encode semantic structure in high-dimensional continuous spaces whose coordinates are difficult to interpret. We introduce Interpretable Sparse Binary Coding (ISBC), which learns a small set of binary dimensions from frozen embeddings without taxonomy labels or downstream outcomes. A trainable encoder and affine decoder combine reconstruction with controls on activity, factor utilization, dependence, and activation uncertainty. Individual bits and their conjunctions select overlapping document groups; complete activation profiles provide finer contexts for conditional comparisons. We evaluate these representations across corpora, backbones, and code configurations through reconstruction, geometry, predictive probes, and document inspection. Empirical cases identify both distinctions within existing categories and shared content across their boundaries. In a mixed-domain corpus, the most frequent profiles predominantly identify source collections, while selected within-source comparisons reveal sentiment and migration contrasts. Selected associations persist in additional documents encoded without refitting. In an engagement application, bits alone improve prediction of log-transformed outcomes over a constant reference, while their combinations identify within-event content contrasts associated with engagement. The results support using binary coordinates and their combinations for empirical interpretation, with complete profiles providing a fine, partly initialization-dependent description rather than a unique discrete taxonomy.
Figures & tables
| Characteristic | Bit 08 inactive | Bit 08 active |
|---|---|---|
| Documents | 1,050 | 1,224 |
| Sports (%) | 97.43 | 98.69 |
| Touchdown or quarterback (%) | 1.62 | 45.75 |
| Home run or inning (%) | 23.90 | 1.47 |
| 8 | 4 | 3.96 | 6.047 | 6.048 | 0.106 | 0.107 | 0.166 |
|---|---|---|---|---|---|---|---|
| 8 | 6 | 6.18 | 4.397 | 4.397 | 0.098 | 0.102 | 0.166 |
| 16 | 6 | 5.94 | 11.322 | 11.391 | 0.160 | 0.161 | 0.245 |
| 32 | 6 | 6.07 | 14.197 | 14.471 | 0.214 | 0.216 | 0.359 |
| 32 | 8 | 8.21 | 15.606 | 16.058 | 0.223 | 0.225 | 0.359 |
| 32 | 16 | 16.05 | 16.452 | 17.049 | 0.232 | 0.234 | 0.359 |
| NN: cosine | NN: Euclidean | Random agreement | Shared-profile coverage | |
|---|---|---|---|---|
| 8/4 | 58.20 | 57.20 | 2.8374 | 100.00 |
| 8/6 | 74.95 | 74.00 | 7.2928 | 100.00 |
| 16/6 | 31.70 | 31.75 | 0.1470 | 96.75 |
| 32/6 | 22.85 | 22.60 | 0.0242 | 75.60 |
| 32/8 | 13.95 | 13.95 | 0.0079 | 49.75 |
| 32/16 | 8.95 | 8.80 | 0.0014 | 24.80 |
| Backbone | Architecture family | Parameters (M) | Effective profiles | Relative BA | Relative reconstruction | ||
|---|---|---|---|---|---|---|---|
| Qwen | Qwen 3 | 600 | 1024 | 5.94 | 2560.2 | 93.99 | 65.23 |
| MiniLM | BERT-style | 22.7 | 384 | 5.97 | 6488.6 | 92.38 | 67.69 |
| NVIDIA | Llama 3.2 | 3200 | 3072 | 5.90 | 521.3 | 95.24 | 70.73 |
| Harrier | Gemma 3 | 270 | 640 | 5.95 | 824.0 | 94.61 | 61.73 |
| EmbeddingGemma 2 | Gemma 4 | 270 | 768 | 6.02 | 1252.4 | 93.19 | 52.87 |
| Community | Active coordinates | Inactive coordinates | Documents | Largest category (%) |
|---|---|---|---|---|
| C6 | G5, H14, N0, Q1, Q9 | M10 | 6,456 | Sports (96.1) |
| C6 | G5, H14, M10, Q9 | N0, Q1 | 1,646 | World (95.6) |
| C10 | G14, M15, Q10 | H13, N5 | 2,969 | Business (85.5) |
| C10 | G14, H13, N5, Q10 | M15 | 9,171 | Sci_Tech (90.2) |
| Without decoder refit | Refitted affine decoder | |||
|---|---|---|---|---|
| Method | Input | NMSE | ||
| ISBC | Binary | 0.840 | 0.160 | 0.161 |
| ITQ top-6 | Binary | 0.970 | 0.030 | 0.152 |
| ITQ | Binary | 0.966 | 0.034 | 0.172 |
| HypotheSAEs | Continuous | 0.784 | 0.216 | 0.141 |
| BinarySentEmb top-6 | Binary | 1.040 | -0.040 | 0.046 |
| Method | Activity (mean SD) | Profiles | Constant bits | ||
|---|---|---|---|---|---|
| ISBC | 12,174 | 11.322 | 2,560.2 | 0 | |
| ITQ top-6 | 6,505 | 10.939 | 1,962.9 | 0 | |
| ITQ | 22,805 | 12.581 | 6,126.4 | 0 | |
| HypotheSAEs | 7,486 | 11.877 | 3,761.2 | 0 | |
| BinarySentEmb top-6 | 20 | 2.614 | 6.1 | 8 | |
| BinarySentEmb | 1 | 0.000 | 1.0 | 16 |
| Corpus | Observed profiles | Top ten (%) | |||
|---|---|---|---|---|---|
| PHEME | 5,802 | 6.75 | 303 | 41.9 | 58.74 |
| Airline | 14,485 | 6.31 | 768 | 87.7 | 43.09 |
| Climate Migration | 5,545 | 6.06 | 780 | 144.6 | 39.62 |
| TweetEval hate | 11,961 | 6.01 | 1,023 | 164.4 | 34.40 |
| 20 Newsgroups | 18,221 | 6.04 | 2,075 | 285.9 | 33.90 |
| TweetEval emotion | 4,678 | 6.04 | 1,089 | 348.0 | 23.75 |
| Corpus | Attribute | AMI | BA [95% CI] | Macro-F1 | ||
| AG News | Topic | 127,600 | 4 | 0.216 | 85.62 [85.19, 86.04] | 85.57 |
| 20 Newsgroups | Topic group | 18,221 | 5 | 0.285 | 80.70 [79.11, 82.29] | 78.09 |
| Topic | 18,221 | 20 | 0.357 | 49.08 [47.65, 50.51] | 47.14 | |
| Airline | Airline | 14,485 | 6 | 0.509 | 97.60 [97.00, 98.15] | 96.91 |
| Sentiment | 14,485 | 3 | 0.057 | 54.39 [52.51, 56.24] | 48.53 | |
| Sentiment airline | 14,485 | 18 | 0.476 | 51.04 [48.80, 53.32] | 44.73 |
| Corpus | Bit | Fixed context | Attribute | (%) | (%) | ||
|---|---|---|---|---|---|---|---|
| Airline | 01 | 1*11110000001000 | 654 | 742 | Negative sentiment | 20.80 | 39.49 |
| Airline | 14 | 00101000000011*1 | 1,064 | 156 | Negative sentiment | 46.52 | 81.41 |
| 20 Newsgroups | 03 | 000*011110001010 | 754 | 645 | comp.windows.x | 32.89 | 64.19 |
| Climate Migration | 09 | 010000110*010011 | 115 | 166 | Migration relevance | 58.26 | 86.14 |
| PHEME | 07 | 1000100*10001111 | 257 | 337 | Rumour | 19.46 | 54.30 |
| Profile | Documents | Dominant source | Source (%) | Recorded attribute | Attribute (%) |
|---|---|---|---|---|---|
| P01 | 3,412 | Airline | 99.68 | Negative sentiment | 76.77 |
| P02 | 1,950 | Climate Migration | 97.44 | Migration relevance | 87.58 |
| P03 | 1,946 | 20 Newsgroups | 98.72 | Computing | 82.09 |
| P04 | 1,794 | FakeNewsNet | 98.33 | Real label | 62.02 |
| P05 | 785 | AG News | 63.69 | Sports | 98.80 |
| P06 | 708 | FakeNewsNet | 97.32 | Real label | 86.50 |
| Context and annotation | Varying bit | Reference | Reference (%) | New | New (%) [95% CI] |
|---|---|---|---|---|---|
| Airline: negative sentiment | 6,266 | 71.0 | 320 | 68.1 [62.8, 73.0] | |
| Bits active | 1,679 | 38.5 | 78 | 39.7 [29.6, 50.8] | |
| Climate Migration: migration | 866 | 55.9 | 79 | 62.0 [51.0, 71.9] | |
| Bits active | 3,688 | 82.4 | 378 | 80.4 [76.1, 84.1] |
| Outcome | Predictors | OLS MSE | RF MSE |
|---|---|---|---|
| Retweets | Constant reference | 1.078 | 1.078 |
| Bits only | 1.020 | 1.022 | |
| Bits + controls | 0.978 | 0.949 | |
| Favorites | Constant reference | 1.059 | 1.059 |
| Bits only | 0.963 | 0.962 | |
| Bits + controls | 0.927 | 0.805 |
| Training | Test | |||
| Documents | 375 | 600 | 99 | 147 |
| Incident expressions (%) | 18.4 | 82.0 | 14.1 | 76.2 |
| Religion/public-response expressions (%) | 52.5 | 3.8 | 43.4 | 6.8 |
| Median retweet count | 193 | 179 | 164 | 178 |
| Median favorite count | 131 | 63 | 105 | 53 |
Appendix figures & tables65 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Hidden layer | 512 units, GELU, preceded by layer normalization |
| Optimizer | AdamW, , , |
| Peak learning rate; weight decay | ; |
| Learning-rate schedule | Linear warm-up over 6% of steps, then linear decay to zero |
| Training budget; gradient clipping | 3 epochs; global gradient norm capped at 1 |
| Trainable parameter precision | FP32 |
| Backbone | Range across configurations | ||
|---|---|---|---|
| MiniLM | 384 | 9.26 | 9.13–9.43 |
| Harrier | 640 | 10.22 | 10.09–10.30 |
| EmbeddingGemma | 768 | 10.28 | 10.14–10.39 |
| Qwen | 1024 | 10.93 | 10.79–10.95 |
| NVIDIA | 3072 | 11.42 | 11.42–11.59 |
| Corpus | Documents | Recorded label dimensions |
|---|---|---|
| AG News | 127,600 | News topic (4) |
| 20 Newsgroups | 18,221 | Newsgroup (20); derived topic group (5) |
| Airline | 14,485 | Sentiment (3); complaint-based refinement (12); airline (6); sentiment–airline combination (18) |
| GoEmotions | 45,432 | Emotion or neutral (28); grouped affect (4) |
| TweetEval sentiment | 57,899 | Negative, neutral, positive (3) |
| TweetEval emotion | 4,678 | Anger, joy, optimism, sadness (4) |
| Sci_Tech -dominant document group | World -dominant document group | ||||
|---|---|---|---|---|---|
| Term | Non- Business | Business | Term | Non- Business | Business |
| line | 3.7 | 0.0 | minister | 14.8 | 0.0 |
| week | 6.3 | 0.7 | peace | 14.4 | 0.0 |
| mp3 | 3.6 | 0.0 | president | 14.0 | 0.0 |
| advanced | 3.0 | 0.0 | talks | 12.9 | 0.0 |
| model | 2.6 | 0.0 | prime | 11.0 | 0.0 |
| Corpus | Attribute | AMI | BA [95% CI] | Macro-F1 | ||
|---|---|---|---|---|---|---|
| Climate Migration | Driver combinations | 5,516 | 53 (40) | 0.123 | 6.42 [5.41, 7.66] | 0.99 |
| Hazard combinations | 5,521 | 27 (21) | 0.015 | 13.12 [9.57, 16.81] | 2.93 |
| First seed | Second seed | Mean matched-bit NMI |
|---|---|---|
| 1234 | 2025 | 0.163 |
| 1234 | 31415 | 0.163 |
| 2025 | 31415 | 0.173 |
| Representation | |||
|---|---|---|---|
| ISBC | 0.578 | 0.550 | 0.489 |
| ITQ top-6 | 0.656 | 0.625 | 0.569 |
| ITQ | 0.620 | 0.590 | 0.530 |
| Technology-market | Business – World | Three-bit coverage | ||||
|---|---|---|---|---|---|---|
| Observed | Random | Observed | Random | Observed | Random | |
| 32 | 9 | 30.8 | 9 | 15.3 | 6 | 21.3 |
| 64 | 20 | 54.3 | 11 | 18.2 | 11 | 41.3 |
| 256 | 39 | 105.3 | 17 | 20.9 | 30.5 | 149.8 |
| Cosine–Cosine | Cosine–Jaccard | Euclidean–Hamming | Euclidean–Jaccard | ||
|---|---|---|---|---|---|
| 8 | 4 | 0.449 | 0.449 | 0.291 | 0.290 |
| 8 | 6 | 0.453 | 0.452 | 0.335 | 0.336 |
| 16 | 6 | 0.494 | 0.493 | 0.330 | 0.311 |
| 32 | 6 | 0.495 | 0.495 | 0.356 | 0.320 |
| 32 | 8 | 0.496 | 0.495 | 0.374 | 0.312 |
| 32 | 16 | 0.501 | 0.500 | 0.344 | 0.333 |
| Representation | BA [95% CI] | Relative BA | Macro-F1 | ||
|---|---|---|---|---|---|
| Original embeddings | – | – | 91.09 [90.76, 91.44] | 100.00 | 91.08 |
| Binary code | 8 | 4 | 85.17 [84.74, 85.59] | 93.50 | 85.10 |
| Binary code | 8 | 6 | 86.79 [86.38, 87.19] | 95.28 | 86.73 |
| Binary code | 16 | 6 | 85.62 [85.19, 86.04] | 93.99 | 85.57 |
| Binary code | 32 | 6 | 87.63 [87.23, 88.02] | 96.20 | 87.60 |
| Binary code | 32 | 8 | 86.94 [86.54, 87.35] | 95.44 | 86.91 |
| Objective | OLS MSE/ | Mean bit entropy | |||
|---|---|---|---|---|---|
| Full | |||||
| Without activity | |||||
| Without balance | – | – | |||
| Without decorrelation | |||||
| Without entropy | |||||
| Without reconstruction |
| Removed term | Diagnostic | Full | Ablated | Paired change |
|---|---|---|---|---|
| Activity control | Absolute activity error | |||
| Balance | Prevalence SD | |||
| Decorrelation | Absolute correlation | |||
| Entropy | Conditional entropy (nats) | |||
| Reconstruction | Decoder MSE/ |
| Objective | Binary BA | binary BA | Continuous BA | Effective profiles |
|---|---|---|---|---|
| Full | ||||
| Without activity control | ||||
| Without balance | ||||
| Without decorrelation | ||||
| Without entropy | ||||
| Without reconstruction |
| Embedding BA | Binary BA | PCA reconstruction | Binary reconstruction | |||||
|---|---|---|---|---|---|---|---|---|
| Pearson | Spearman | Pearson | Spearman | Pearson | Spearman | Pearson | Spearman | |
| 0.793 | 0.667 | 0.849 | 0.821 | 0.280 | 0.103 | 0.447 | 0.359 | |
| Embedding dimension | 0.616 | 0.600 | 0.688 | 0.700 | 0.386 | 0.300 | 0.787 | 0.500 |
| Backbone | Observed profiles | Mean documents per profile | Documents in singleton profiles (%) |
|---|---|---|---|
| Qwen | 12,174 | 10.48 | 3.48 |
| MiniLM | 20,470 | 6.23 | 6.11 |
| NVIDIA | 4,862 | 26.24 | 1.26 |
| Harrier | 5,462 | 23.36 | 1.31 |
| EmbeddingGemma 2 | 6,490 | 19.66 | 1.54 |
| Backbone 1 | Backbone 2 | ARI | AMI |
|---|---|---|---|
| Qwen | MiniLM | 0.095 | 0.233 |
| Qwen | NVIDIA | 0.106 | 0.376 |
| Qwen | Harrier | 0.087 | 0.284 |
| Qwen | EmbeddingGemma 2 | 0.059 | 0.221 |
| MiniLM | NVIDIA | 0.077 | 0.295 |
| MiniLM | Harrier | 0.065 | 0.213 |
| Rank | Reference documents | First matched backbone | Initial overlap (%) | Five-profile intersection |
| 1 | 1375 | Harrier | 19 | 2 |
| 2 | 1224 | NVIDIA | 90 | 68 |
| 3 | 1075 | Harrier | 70 | 69 |
| 4 | 1066 | NVIDIA | 60 | 56 |
| Community/node | Documents | Coverage (%) | First | Second | Third | Fourth |
|---|---|---|---|---|---|---|
| C1 | 104,797 | 82.1 | World (30.0) | Sci_Tech (27.6) | Business (26.8) | Sports (15.6) |
| C2 | 80,745 | 63.3 | Sports (39.5) | Business (25.2) | Sci_Tech (24.5) | World (10.8) |
| C3 | 82,792 | 64.9 | Business (36.9) | Sci_Tech (31.9) | World (20.0) | Sports (11.2) |
| C4 | 100,368 | 78.7 | World (28.1) | Business (25.5) | Sports (25.1) | Sci_Tech (21.4) |
| C5 | 97,065 | 76.1 | Sports (28.4) | Business (28.3) | Sci_Tech (27.2) | World (16.1) |
| C6 | 64,187 | 50.3 | Sports (47.4) | World (39.9) | Business (7.0) | Sci_Tech (5.6) |
| Reference | Profiles | Coverage (%) | Final documents | Retained (%) | Category coherence | Chain null |
|---|---|---|---|---|---|---|
| ISBC | 10 | 8.06 | 295 | 2.87 | 0.732 | 0.608 |
| HypotheSAEs | 10 | 2.49 | 301 | 9.49 | 0.758 | 0.603 |
| BinarySentEmb top-6 | 10 | 99.54 | 271 | 0.21 | 0.708 | 0.582 |
| BinarySentEmb | 1 | 100.00 | 45 | 0.04 | 0.685 | 0.650 |
| Representation | BA [95% CI] | Macro-F1 | Relative BA |
|---|---|---|---|
| ISBC, binary | 85.62 [85.20, 86.06] | 85.57 | 93.97 |
| ITQ, top-6 | 82.58 [82.14, 83.03] | 82.53 | 90.64 |
| ITQ, binary | 84.82 [84.36, 85.26] | 84.79 | 93.10 |
| HypotheSAEs, support | 75.37 [74.89, 75.89] | 75.10 | 82.72 |
| BinarySentEmb, top-6 | 50.40 [49.81, 50.94] | 49.17 | 55.32 |
| BinarySentEmb, binary | 25.00 [25.00, 25.00] | 10.00 | 27.44 |
| Representation | Mean activity | BA [95% CI] | Macro-F1 | |
|---|---|---|---|---|
| ISBC, binary | 5.94 | 85.62 [85.20, 86.06] | 85.57 | 0.161 |
| ITQ, top-6 | 6.00 | 82.58 [82.14, 83.03] | 82.53 | 0.152 |
| ITQ, binary | 7.98 | 84.82 [84.36, 85.26] | 84.79 | 0.172 |
| PCA-sign | 7.98 | 82.14 [81.66, 82.59] | 81.99 | 0.160 |
| PCA top-6 | 6.00 | 79.39 [78.89, 79.86] | 79.22 | 0.144 |
| Random sign | 7.99 | 54.77 [54.15, 55.36] | 54.49 | 0.069 |
| Target | Classes | AMI | BA [95% CI] | Macro-F1 |
|---|---|---|---|---|
| Source collection | 9 | 0.236 | 71.35 [70.79, 71.90] | 67.43 |
| Document type | 5 | 0.160 | 78.18 [77.62, 78.76] | 73.45 |
| P01: bit 11 inactive | P07: bit 11 active | |||
|---|---|---|---|---|
| Carrier | Documents | Negative (%) | Documents | Negative (%) |
| American | 929 | 76.32 | 117 | 54.70 |
| Delta | 353 | 63.17 | 32 | 28.12 |
| Southwest | 581 | 71.94 | 232 | 42.24 |
| US Airways | 1,018 | 81.63 | 87 | 60.92 |
| United | 456 | 84.87 | 26 | 73.08 |
| Sampling component | New documents | Mean active bits | Observed reference profile (%) |
|---|---|---|---|
| 20 Newsgroups | 500 | 6.26 | 94.4 |
| AG News | 500 | 5.82 | 95.6 |
| Airline | 500 | 6.32 | 96.6 |
| Climate Migration | 500 | 6.81 | 99.0 |
| FakeNewsNet | 500 | 5.74 | 97.0 |
| GoEmotions | 500 | 5.78 | 94.4 |
| Retweets | Favorites | |||
|---|---|---|---|---|
| Predictor specification | OLS | RF | OLS | RF |
| Constant reference | 1.078 | 1.078 | 1.059 | 1.059 |
| Bits only | 1.020 | 1.022 | 0.963 | 0.962 |
| Bits + controls | 0.978 | 0.949 | 0.927 | 0.805 |
| Veracity | 1.033 | 1.090 | 0.960 | 0.939 |
| Veracity + bits | 0.978 | 0.936 | 0.896 | 0.792 |
| All | Charlie Hebdo | Ferguson | Germanwings | Ottawa | Sydney | |
|---|---|---|---|---|---|---|
| 2,673 | 269 | 1,142 | 7 | 40 | 1,215 | |
| 782 | 312 | 1 | 462 | 2 | 5 | |
| 871 | 22 | 0 | 0 | 848 | 1 | |
| 1,476 | 1,476 | 0 | 0 | 0 | 0 |
| Term | ( ) | ( ) |
|---|---|---|
| jesuischarlie | 79 (10.7%) | 218 (29.5%) |
| cartoons | 13 (1.8%) | 68 (9.2%) |
| freedom | 25 (3.4%) | 56 (7.6%) |
| solidarity | 15 (2.0%) | 38 (5.1%) |
| attack | 259 (35.1%) | 84 (11.4%) |
| police | 119 (16.1%) | 3 (0.4%) |
| , all | Median favorites | Lower 10% | Upper 10% | |
|---|---|---|---|---|
| 10/269 | 42 | 3/63 (4.8%) | 1/10 (10.0%) | |
| 197/312 | 85 | 6/17 (35.3%) | 23/36 (63.9%) | |
| 1/22 | 43.5 | 0/4 (0.0%) | 0/3 (0.0%) | |
| 739/1476 | 88 | 21/127 (16.5%) | 99/159 (62.3%) |
| Event | Outcome | Bit | Lower 10% | Upper 10% | (pp) |
|---|---|---|---|---|---|
| Sydney | Favorites | 07 | 104/125 (83.2%) | 43/123 (35.0%) | -42.0 |
| Ottawa | Favorites | 14 | 75/93 (80.6%) | 57/89 (64.0%) | -13.8 |
| Ferguson | Retweets | 07 | 76/115 (66.1%) | 91/116 (78.4%) | +12.6 |
| Ferguson | Favorites | 07 | 95/116 (81.9%) | 81/115 (70.4%) | -9.8 |
| Outcome | Active bits | Tail (%) | Cosine L | Cosine U | ||||
|---|---|---|---|---|---|---|---|---|
| Retweets | 10 | 113 | 124 | 3.668 | 3.291 | 0.455 | 0.474 | |
| Retweets | 25 | 273 | 302 | 3.631 | 3.484 | 0.452 | 0.470 | |
| Favorites | 10 | 176 | 100 | 3.211 | 3.006 | 0.471 | 0.461 | |
| Favorites | 25 | 435 | 265 | 3.277 | 3.120 | 0.464 | 0.451 | |
| Favorites | 10 | 21 | 99 | 0.792 | 0.244 | 0.556 | 0.535 | |
| Favorites | 25 | 72 | 241 | 0.884 | 0.490 | 0.558 | 0.536 |
Sparse Coding
Sparse coding represents data as combinations of a small number of learned or structured dictionary elements.
See all 54 papers