cs.LGSep 3, 2026
SaveSharedSAE: One Feature Dictionary Across Language Models
Organizations: MBZUAI · Sorbonne Université · Tohoku University · RIKEN AIP
Abstract
Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.
Figures & tables
Figure 1: SharedSAE replaces post-hoc matching with one persistent, adaptable dictionary. Top left: dedicated SAEs admit only partial post-hoc matching of their latents. Top right: SharedSAE trains per-model encoders and decoders around one shared dictionary and later attaches a new model with the dictionary frozen. Bottom left: the criteria a shared SAE must meet, with each method’s score. Bottom right: under solo TopK, SharedSAE ’s latents correlate across models far more strongly than post-hoc matched SAEs, USAE, or SPARC.
Figure 2: SharedSAE. The aligner cuts each model’s token stream into windows that decode to the same text and reads at the last token of each window (highlighted). Per-model encoders and decoders surround one shared dictionary: the codes are normalized and summed, one TopK gives the index set for all models, and with probability a single random model picks alone (dice), which trains solo inference. Each decoder reconstructs its own model (self loss) and every other model from that model’s rescaled code.
Figure 3: How many models use a latent: - 4 when all four models select it equally often, 1 for single-model use.
Figure 4: The share of a latent’s code mass one model contributes within a pair, pooled over the six pairs and symmetrized.
Figure 5: Latent selection under the solo TopK, per token: each model’s encoder alone selects its top-128 latents for the same aligned token; the distributions count how many of those latents all four models select at once. Dotted lines mark the means.
Figure 6: Latent selection under the solo TopK, per latent: we obtain the top-50 tokens that activate each latent and report the four-model intersection of these four sets.
Figure 7: Per-latent cross-model value correlation under the joint TopK (top), where all four models vote for the latents to use, and under the solo TopK (bottom), where each model chooses alone. The labels are each system’s median
| Qwen3 | Llama-3.2 | OLMo-2 | Gemma-3 | Mean | Cross-FVE | |
| Reconstruction (FVE) | ||||||
| Dedicated (ceiling) | 0.838 | 0.807 | 0.812 | 0.868 | 0.831 | - |
| SharedSAE , joint TopK | 0.798 | 0.785 | 0.787 | 0.825 | 0.799 | 0.749 |
| SharedSAE , solo TopK | 0.804 | 0.794 | 0.795 | 0.821 | 0.803 | - |
| USAE | 0.760 | 0.743 | 0.750 | 0.787 | 0.760 | 0.640 |
| SPARC, joint TopK | 0.769 | 0.765 | 0.763 | 0.631 | 0.732 | 0.663 |
Table 1: Held-out reconstruction and spliced behavior, every system at its own converged budget, under the joint TopK and the solo TopK. For CE-recovered and KL, the reconstruction replaces the residual stream and next-token behavior is scored. SPARC’s codes are unit-norm, so its printed reconstruction uses the oracle per-token rescale into each model’s space; with the deployable one-scalar-per-model median-norm rescale its means drop to 0.643 (joint) and 0.490 (solo). Blanks are due to architectural differences: USAE and the crosscoder have no joint/solo split, the crosscoder has no cross-reconstruction. The dedicated rows are the per-model ceiling, a best-case reference. Bold marks the best shared system per column within each block excluding the dedicated reference.
| decoded into model | ||||
|---|---|---|---|---|
| code from | Qwen3 | Llama-3.2 | OLMo-2 | Gemma-3 |
| Qwen3 | 0.798 | 0.738 | 0.738 | 0.782 |
| Llama-3.2 | 0.746 | 0.785 | 0.738 | 0.786 |
| OLMo-2 | 0.741 | 0.733 | 0.787 | 0.780 |
| Gemma-3 | 0.739 | 0.733 | 0.736 | 0.825 |
| Dedicated | 0.815 | 0.783 | 0.795 | 0.825 |
Table 2: Self- and cross-reconstruction FVE of SharedSAE under the joint TopK.
| Clarity | Resp. | Purity | Faith. | Retained | |
| Within-stream clear labels (clarity ) | |||||
| SharedSAE shared vote | 0.979 | 0.642 | 0.463 | 0.281 | 10.6% |
| SharedSAE per-stream | 0.976 | 0.629 | 0.438 | 0.217 | 21.7% |
| SAE Qwen3 | 0.975 | 0.584 | 0.416 | 0.333 | 7.4% |
| SAE Llama-3.2 | 0.962 | 0.558 | 0.431 | 0.403 | 9.4% |
| SAE OLMo-2 | 0.954 | 0.583 | 0.373 | 0.320 | 4.1% |
Table 3: FADE evaluation of SharedSAE and dedicated SAE latents. Top: within-stream evaluation of label–stream pairs satisfying clarity . Retained reports the fraction of evaluated label–stream pairs satisfying the clarity threshold. Middle: cross-stream transfer of source-validated labels. Shared-vote labels are retained if they reach clarity in at least one stream and are then evaluated on all streams. Clearest per-stream selects, for each latent, the per-stream label with highest source-stream clarity and retains it when this value exceeds . Bottom: no clarity filtering is applied. For each latent, the per-stream label with highest source-stream clarity is selected and FADE scores are correlated across the six model pairs.
Figure 8: Most-activating tokens for selected single-concept latents, across the four models SharedSAE was trained on and two models adapted to the frozen dictionary afterwards. min effect is the probe gap in the weakest model, in standard-deviation units. Full contexts are in Appendix O .
| Seed TopK | Solo | ||||
| Held-out model | FVE | /co-tr. | /ded. | FVE | /ded. |
| Leave-one-out (three-model seed) | |||||
| Qwen3 | 0.769 | 0.997 | 0.944 | 0.793 | 0.972 |
| Llama-3.2 | 0.753 | 0.990 | 0.962 | 0.769 | 0.982 |
| OLMo-2 | 0.751 | 0.984 | 0.945 | 0.776 | 0.977 |
| Gemma-3 | 0.787 | 0.979 | 0.954 | 0.812 | 0.984 |
Table 4: Adapting a held-out model to a frozen SharedSAE : only the new model’s encoder, decoder, and biases train, so the seed models and their latent descriptions stay untouched.
| corr. | top-50 | choice | FVE/ded. | |
| Co-trained SharedSAE (4 models) | 0.785 | 31.5 | 59.0 | 0.999 |
| Adapted, leave-one-out mean | 0.785 | 31.5 | 66.0 | 0.979 |
| Held-out Gemma fold, attach routes | ||||
| adapted encoder-decoder (ours) | 0.785 | 32.0 | 60.3 | 0.984 |
| linear alignment (Atlas-style) | 0.722 | 29.5 | 57.5 | 0.692 |
| decoder-only attach | - | - | - | 0.795 |
Table 5: The Section 4 criteria under adaptation, with the adapted model voting solo: median per-latent value correlation and top-50 overlap against the seed models, mean per-token intersection of the solo selections (of 128; four-way above, five-way in the Qwen3.5 and Instruct blocks), and solo FVE as a share of the dedicated SAE.
Figure 9: Validation FVE by epoch at the 4M protocol: dedicated SAEs (left), SharedSAE self-reconstruction (middle), and SharedSAE cross-reconstruction (right, colored by target). Every curve rises monotonically through epoch 8 (dashed line), so the dedicated ceilings and SharedSAE train on to convergence (Section A ). The middle and right panels add the converged SharedSAE run’s logged tail, epochs 41-62, flat at its early stop; a node requeue truncated that run’s earlier log, and the probe stream drawn for epochs 1-8 is a separate run (Appendix C ).
| Vote input (10 epochs, no dropout) | |||
|---|---|---|---|
| Share | Vote shares (Q/L/O/G) | ||
| raw votes | 0.919 | 0.07/0.00/0.00/0.93 | |
| RMS-cal. inputs | 0.925 | 0.24/0.40/0.30/0.07 | |
| vote norm (raw) | 0.930 | 0.25/0.25/0.25/0.25 | |
| Vote-source dropout dose (vote norm) | |||
| Share | Cross-FVE | Gemma solo | |
Table 6: Ablations against the settled configuration (vote norm, vote-source dropout , oracle per-token cross rescale, ); each block varies one ingredient with the rest held fixed. Share, vote shares (Q/L/O/G), Corr., and Top-50 are defined in Appendix C . Within each block, the best value per column is in bold. Measured at the earlier protocol (800k-position store, , 10 epochs, held-out shard 3), not rerun at the 4M protocol.
| CE-recovered | ||||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3 | Llama | OLMo-2 | Gemma | Qwen3 | Llama | OLMo-2 | Gemma | |
| Dedicated (8-epoch) | 0.966 | 0.922 | 0.965 | 0.943 | 0.243 | 0.484 | 0.309 | 0.423 |
| SharedSAE (settled) | 0.958 | 0.933 | 0.970 | 0.968 | 0.309 | 0.438 | 0.279 | 0.293 |
Table 7: Functional splice at the hook layer under the solo TopK: the reconstruction replaces the residual stream and next-token behavior is scored, per stream, against the budget-matched 8-epoch dedicated SAEs (the converged dedicated splice is in Table 1 ). CE-recovered and KL are defined in Appendix C . Best value per column in bold (higher CE-recovered, lower KL).
| Qwen3 | Llama-3.2 | OLMo-2 | Gemma-3 | Mean | |
| SharedSAE (mean consensus size of 128) | |||||
| all own 128 latents | 0.804 | 0.794 | 0.795 | 0.821 | 0.803 |
| consensus latents only | 0.736 | 0.721 | 0.727 | 0.716 | 0.725 |
| own strongest latents | 0.758 | 0.740 | 0.746 | 0.739 | 0.746 |
| latents outside the consensus | -1.139 | -0.010 | 0.032 | -8.247 | -2.341 |
| USAE, 32 epochs (mean consensus size of 128) | |||||
Table 8: Reconstruction (FVE) through the consensus latents. For each token, each model’s encoder alone selects its top-128 latents under the solo TopK; the consensus set is the intersection of the four selections. Each model is then reconstructed from its own code restricted to: all 128 own selections; the consensus set only; its own strongest latents, a size-matched control; and its selections outside the consensus.
| 100k window, latents alive 10 times | ||
|---|---|---|
| Pairwise (of 50) | All-four | |
| System | median (span) | (of 50) |
| SharedSAE | 37 (37-38) | 26 |
| Matched dedicated | 15 (11-17) | - |
| USAE (8 epochs) | 0 (0-0) | - |
| 300k window, latents with 50 firings ( ) | ||
Table 9: Top-50 activating-position overlap: how many of a latent’s 50 top-activating positions coincide across models. Top block: median over the six model pairs (span in parentheses) and the all-four count on the core of latents alive at least 50 times in every model. Bottom block: the same statistic under each selection condition on the 300k window, with each system’s latent population in parentheses; USAE codes are per-model native, so only the solo column is defined. Chance overlap is 0 of 50. Dashes: statistic undefined or not computed for the system (Appendix C ). Best value per column in bold.
| Vocabulary classes ( usage, 100k-token window) | ||
| Class | Latents | Med. corr. |
| Consensus (all four) | 10,776 | 0.91 |
| Partial (two or three) | 1,650 | 0.85 |
| Specific (one model) | 1,144 | 0.84 |
| Below everywhere | 19,190 | 0.82 |
| Stability of the 20k-window classes at 100k | ||
Table 10: Solo-TopK usage classes at of tokens, the migration of the 20k-window classes when the window grows to 100k tokens, and per-stream statistics. Med. corr. is the median per-latent cross-model code correlation within the class; populations and definitions in Appendix C .
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Dedicated ceilings trained until validation FVE levels off, under the constant-rate recipe and a 64-epoch cap. Bold: the best validation FVE reached so far - the checkpointed ceiling; faint: the per-epoch series. Dots mark each stream’s peak; the cross marks Llama’s early stop (peak unbeaten for eight consecutive epochs); the dashed line is the 8-epoch budget of every shared system. The second Qwen3 seed (dotted) is indistinguishable from the first throughout. This full-shard window sits 0.5-1.0 points of FVE below the evaluation window of Table 1 (Appendix C ).
Figure 11: The post-hoc-matching reference for the second criterion. Per-latent activation correlations: the best post-hoc match between separately trained dedicated SAEs of different models (blue; converged arms, median 0.45), the same matching between two SAEs of the same model trained from different seeds (gray; median 0.77) - the ceiling the post-hoc route permits with no cross-model disagreement at all - and SharedSAE ’s shared-latent correlation with each model voting alone (orange; median 0.79; the dotted tick marks the joint-TopK median, 0.90). Matching cannot reach its own within-model ceiling across models; the shared dictionary clears that ceiling solo and exceeds it further under the joint vote. SharedSAE ’s near-zero mass is rare latents - 3% of measured latents, 0.1% of solo selections (Section 4.3 ). Matching controls are measured on the 100k window and SharedSAE correlations on the 300k window; the within-model arm is one seed pair on one stream (protocols in Appendix C ).
Figure 12: Solo per-latent value-correlation distributions inside each three-model leave-one-out seed (step outlines, colored by the held-out model) over the full four-model SharedSAE (orange fill). The four seed medians coincide with each other and with the full dictionary’s.
| seed | corr. | top-50 | pairwise | -way | FVE/conv. |
|---|---|---|---|---|---|
| without Qwen3-1.7B | 0.778 | 32.0 | 83.3 | 71.8 (3-way) | 0.945 |
| without Llama-3.2-1B | 0.784 | 32.0 | 85.9 | 75.0 (3-way) | 0.942 |
| without OLMo-2-1B | 0.791 | 32.0 | 85.8 | 74.8 (3-way) | 0.935 |
| without Gemma-3-1B | 0.792 | 32.0 | 80.9 | 68.5 (3-way) | 0.956 |
| full SharedSAE | 0.797 | 31.5 | 78.0 | 59.0 (4-way) | 0.967 |
Table 11: Internal sharedness of each three-model leave-one-out seed, members voting solo: pooled per-latent value-correlation median and top-50 overlap median (300k window), mean pairwise and -way per-token selection intersection of 128 (e02 window), and mean solo FVE as a share of the members’ converged dedicated ceilings. The seeds train for 8 epochs, so their shares sit below the converged four-model row by the ceiling gap, not by any property of the roster. The full-model row restates published values.
| under the solo TopK | Qwen3 | Llama | OLMo | Gemma |
|---|---|---|---|---|
| SPARC, vote share | 0.245 | 0.276 | 0.274 | 0.205 |
| SPARC, solo joint | 0.711 | 0.625 | 0.592 | 0.635 |
| SPARC, latents used | 22,933 | 30,635 | 31,097 | 7,719 |
| SharedSAE , latents used | 24,328 | 24,776 | 24,532 | 24,964 |
| Four-way intersection of the solo selections, of 128 | ||||
| SPARC 49.2 SharedSAE 58.9 | ||||
Table 12: SPARC’s solo-TopK condition. Left: selection behavior on the scoring window of Table 1 - vote share is each stream’s mean share of the aggregated selection mass, solo joint the fraction of the joint top- that a stream’s own top- reproduces, and latents used the number of distinct latents a stream’s solo vote ever selects (the joint selection uses 7,599 for SPARC and 23,714 for SharedSAE ). Right: the same condition per model pair - how many of the 128 selections the two streams share at a token, and the two per-latent medians of Table 9 and Figure 7 with the population they are measured over (300k window). Our joint selection was verified to reproduce SPARC’s own shared_indices exactly on every batch.
| model | ded. | joint | share | solo | solo/ded | sink % |
|---|---|---|---|---|---|---|
| Qwen3 | 0.807 | 0.768 | 0.952 | 0.767 | 0.951 | 0.015 |
| Llama-3.2 | 0.782 | 0.738 | 0.944 | 0.753 | 0.963 | 0.000 |
| OLMo-2 | 0.712 | 0.653 | 0.917 | 0.681 | 0.957 | 0.000 |
| Gemma-3 | 0.819 | 0.782 | 0.954 | 0.751 | 0.917 | 0.000 |
Table 13: Mid-depth ( 50%) roster: the settled SharedSAE (joint and solo TopK) against dedicated (ded.) ceilings at hooks qwen3-14 llama-8 olmo2-8 gemma-13 (mid ˜50%). share and solo/ded are ratios to the dedicated FVE, and sink % is defined in Appendix C . Cross-FVE mean is 0.652 (max 0.743). Measured at the earlier protocol (800k-position store, , 10 epochs, held-out shard 3), not rerun at the 4M protocol.
| class | @20k | @100k | corr |
|---|---|---|---|
| consensus (all four) | 9,987 | 10,776 | 0.91 |
| partial (2-3 models) | 1,877 | 1,650 | 0.85 |
| specific (one model) | 1,397 | 1,144 | 0.84 |
| joint-only | 17 | 8 | 0.72 |
| below everywhere | 19,490 | 19,190 | 0.82 |
Table 14: Latent classes by solo-TopK usage (threshold of tokens): latent counts when classified on a 20k-token window vs. the full 100k held-out shard; “corr” is the class’s median cross-model code correlation at 100k. The migration between the two windows is quantified in Table 10 and Appendix I .
| latent | fires/100k | Qwen3-1.7B | Llama-3.2-1B | OLMo-2-1B | Gemma-3-1B |
|---|---|---|---|---|---|
| 8319 | 50 | free | free | free | free |
| 29780 | 52 | i | i | i | i |
| 11723 | 75 | ,’’, ," | ,", ,’’ | ,’’, ," | ,", ,’’ |
| 16628 | 22 | United | United | United | United |
| 30588 | 20 | United | United | United | United |
| 10904 | 56 | bar, Bar | bar, Bar, -bar | bar, Bar | bar, Bar |
Table 15: Rare shared latents: latents selected on 20-99 of 100k held-out tokens whose top-15 activating positions coincide across all four models. Cells show each model’s top activating tokens, decoded by its own tokenizer. Six of the eight sampled latents are shown; both “United” detectors are kept, an instance of feature splitting. Measured at the earlier protocol (800k-position store, , 10 epochs, held-out shard 3), not rerun at the 4M protocol.
| Token level: ten strongest positions decoded in | |||
|---|---|---|---|
| each model’s own tokenizer (medians over 6 pairs) | |||
| SharedSAE | Matched ded. | ||
| Modal decoded-token agreement | 0.63 | 0.45 | |
| Decoded token-set Jaccard | 0.50 | 0.19 | |
| Concept level: AP retention of each model’s | |||
| five most selective detectors | |||
Table 16: Semantic readout under SharedSAE and under the matched dedicated control, at the token level and at the concept level. The AP-in-own-model column gives each detector’s base selectivity; retention is the ratio of a detector’s average precision read out in the receiving model to that base value. Protocol in Appendix C . Bold: higher retention per row; ties at the printed precision are both bolded.
| Latent | Per-stream Qwen3 | Per-stream OLMo-2 | Per-stream Gemma-3 | Per-stream Llama-3.2 | Shared Vote |
|---|---|---|---|---|---|
| 16291 | word ‘in’ or ‘of’ | word ‘of’ or ‘to’ or punctuation marks | word ‘of’ or ‘in’ | word ‘of’ or ‘is’ or ‘in’ | prepositions and articles |
| 2317 | four-year term or similar duration | four-year term or similar duration | year term election context | four-year term or similar duration | four-year term or similar duration |
| 6418 | using or a before handheld gps | handheld gps usage | word-initial ‘a’ or ‘using’ | word-initial ‘a’ or ‘the’ | word-initial ‘a’ or ‘using’ |
| 2727 | word-initial uppercase fragment | word-initial uppercase fragment | word-initial uppercase fragment | word-initial uppercase fragment | word-initial uppercase fragment |
| 14390 | possessive form | possessive form | word-initial uppercase fragment | possessive ‘s or whose | possessive form |
| 22504 | should | should | should fragment | should | should |
Table 17: Examples of shared SharedSAE latents and their per-stream semantic descriptions.
Figure 13: Qualitative steering examples for shared SharedSAE latents activated across all four models. The same shared latent is steered independently in each model, illustrating that semantically aligned latents can induce related behavioral shifts while leaving room for model-specific effects.
| Dedicated baseline | ||
|---|---|---|
| SAE Gemma-3 | -1.154 | 0.259 |
| SAE Llama-3.2 | -1.002 | 0.327 |
| SAE OLMo-2 | -0.210 | 0.840 |
| SAE Qwen3 | -0.420 | 0.679 |
Table 18: Welch two-sample tests comparing FADE faithfulness between SharedSAE shared-vote labels and each dedicated SAE baseline on retained label–stream pairs. None of the comparisons is significant at .
Figure 14: Same latent, same effect: for each of the 280 mined candidate latents, the overlap of the top-30 tokens the latent promotes across the four models (median over the six pairs), measured causally by injection at the hook layer (filled) and by the linear logit lens (outline). A mismatched-latent null sits at 0.00. String-level overlap is a lower bound: the same concept often surfaces in different languages and subword forms across tokenizers (Appendix N ).
Figure 15: Cross-model steering dose–response, mean over the 12 steered latents. Top: concept-rate lift over each dose regime’s own unsteered baseline; the dotted line is the matched-norm random-latent control. Bottom: judged coherence; arrowheads mark unsteered baselines. Doses left of the dashed line use repetition-penalized sampling, right of it plain sampling. The three width-2048 streams move nearly in lockstep; Gemma-3’s sampler bounds its judged coherence at every arm.
| #1537 - positive appraisal (English and French probes) |
|---|
| Qwen3 |
| (r1, in 4/4 top-10) …ists. |The spine and ribs ofgnathodon saturator| It’s a very nice display. And while the amount … † |
| (r5, in 4/4 top-10) …hem in a history that is both analytical and narrative…A superb book that will be a standard m … † |
| (r10, in 4/4 top-10) …inifa should get an Israel Business award! What a Kiddushem! Nice to hear about the positive asp … † |
| Llama-3.2 |
| (r1, in 4/4 top-10) …ists. |The spine and ribs ofgnathodon saturator| It’s a very nice display. And while the amount … † |
Table 19: Latent 1537: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 38–43 distinct passages per model.
| #7393 - general negative affect (moral-wrongdoing and distress probes) |
|---|
| Qwen3 |
| (r1, in 4/4 top-10) But the power it would need to guarantee that no one is poor would be so great it could crush the natural rights and liberty of individuals. |
| (r5, in 3/4 top-10) Smederevo citizens, frightened and hurt by a harsh occupation, and on April 6, by the bombing of the city, too, withdrew to their homes and only came out on the streets when needed: to finish their jobs and to procure food. |
| (r10, in 3/4 top-10) Those disciples of Christ who have experienced their status to any degree have become keenly aware of the intimate relationship they have with Marya relationship that at the moment is perhaps closer than that with Christ. For it is Mary who brings us to Christ. |
| Llama-3.2 |
| (r1, in 4/4 top-10) He concludes-echoing historian Rick Atkinson’s excellent recent account of the campaign, The Day of Battle-that despite its terrible cost, the fight in Italy played a decisive role in defeating Germany. |
Table 20: Latent 7393: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 30–32 distinct passages per model.
| #27599 - epistemic hedging: conjecture and possibility |
|---|
| Qwen3 |
| (r1, in 4/4 top-10) All three were calibrated using the same unit of measurement, the cubit; the Egyptians broke the cubit into smaller units, which allowed them to keep remarkably accurate records, perhaps more accurate than would have been warranted for the purposes of merely agriculture and taxation. |
| (r5, in 4/4 top-10) But, Namibian elephants are of great interest to tourists, and this may be the key to their salvation in this country that has been described as the land that God created in anger. |
| (r10, in 2/4 top-10) But, Namibian elephants are of great interest to tourists, and this may be the key to their salvation in this country that has been described as the land that God created in anger. |
| Llama-3.2 |
| (r1, in 4/4 top-10) The American Blacks played a crucial role in the very beginning of the Sydney steel plant, but never stayed around to reap any benefits. |
Table 21: Latent 27599: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 35–40 distinct passages per model.
| #23065 - conditional mood in hypothetical clauses |
|---|
| Qwen3 |
| (r1, in 2/4 top-10) Simulations are supporting the education system to better equip students with practical knowledge, so when it comes to tackling real-life situations in their future careers they can perform better. Taking into consideration the benefits offered by simulations in the education process, it would be a shame to miss out on them. |
| (r5, in 3/4 top-10) The feature was named by Blanchard and Davis,66 who called it the Bone Springs arch. It would seem from their paper that they considered the feature to be anticlinal, and to have a similar, opposing flank to the north. |
| (r10, in 3/4 top-10) Thoth - Link With Sirius Sirius Mythology - Zep Tepi Star - Star Tetrahedron Thoth The Scribe used sacred geometry - geometry of the stars to write our program placing coded messages in the heavens for you to find. Creators - Guardians - ’God/Dog’ - Dog Star - would live beyond the stars in a higher realm of consciousness. |
| Llama-3.2 |
| (r1, in 3/4 top-10) Didn’t scientific research recently concluded that when in the dark (can’t be seen!), people are more likely to steal? It would seem that we all agree on the moral high grounds but consistently fail when faced such tests. |
Table 22: Latent 23065: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 40–43 distinct passages per model.
| #20341 - programming operations, in code and in prose about code |
|---|
| Qwen3 |
| (r1, in 4/4 top-10) …ned a new file via my IDE and turned it into a list via read lines . Why? Because I want to † |
| (r5, in 4/4 top-10) …e opened a new file via my IDE and turned it into a list via read lines. Why? Because I want † |
| (r10, in 3/4 top-10) When I use [’Some name__________1.5 6.5 6.5\n’, ’Another name________6.3 1.2 1.5\n’] 1.5 3.5 4.5 2.5 3.5 4.5 5.5 3.5 4.5 fileFolder = open(’TEXTFILE’, ’r’) readFile = fileFolder.readlines () for line in readFile: line = line.split("_") grades = line[-1] grades = map(float,line[-1]) could not convert string to float: . |
| Llama-3.2 |
| (r1, in 4/4 top-10) map is operating on each character of the string, because when you iterate over a string you get individual characters. |
Table 23: Latent 20341: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 8–9 distinct passages per model; this latent concentrates in the window’s few code-bearing documents (largest passage up to 44% of its top 50).
| #30007 - sentence-initial imperatives: calls to action |
|---|
| Qwen3 |
| (r1, in 4/4 top-10) Glyptodonts became extinct at the end of the last ice age along with a large number of other megafaunal species, including pampatheres, the giant ground sloths, and the Macrauchenia. Finding an ancient shell like this is surely not an ordinary discovery. |
| (r5, in 1/4 top-10) The second class is known as “blockchain technology”, and consists of both cryptocurrencies like Bitcoin and Ethereum, and alternative distributed ledger designs like Hyperledger and ErisDB. |
| (r10, in 3/4 top-10) If this ratio is used to design the board the board size would be increased in area by greater than 9 times. |
| Llama-3.2 |
| (r1, in 2/4 top-10) …d her bed, pulling down the covers, and patting the pillow). Give your child some advance notice … † |
Table 24: Latent 30007: examples at activation ranks 1/5/10 per model under its own solo vote; the activating word is bold; ( , in /4 top-10) gives the example’s rank and how many models share the position. Top-50 activations span 38–40 distinct passages per model.
More in this topic
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 776 papers