Backpack language models predict with a strictly linear combination of non-contextual per-word sense vectors, weighted by non-negative contextualization scores. We show that non-negativity conflates two properties: the linear readout, which carries the interpretability, and the conic combination, which merely restricts it. We relax the latter by computing the sense weights with differential attention, a difference of two softmax maps with a learned per-sense gain, so context can actively inhibit a sense; we call the result a differential Backpack. At equal parameters on WikiText-103 this cuts the Backpack's perplexity gap to a Transformer by 31%, improves over the softmax Backpack at every sense count, persists with a shrinking margin up to 350M parameters on OpenWebText, and is localized at the sense layer by ablations and controls. The learned inhibition is directly interpretable (a financial context drives the river sense of bank to -0.50), and the preserved per-sense attribution predicts counterfactual removals exactly. We tested a number of combination rules, showing the linear interface is nearly free. Making the readout nonlinear gains only 0.43 perplexity points while invalidating two thirds of the decomposition. The differential Backpack's senses are also better representations, improving zero-shot sentence similarity by up to 33%.
Figures & tables
Figure 1: The differential Backpack. Non-contextual sense vectors C(xj)ℓ (left) are combined by signed weights Aℓ=A1−λℓA2 (right) under a linear readout; the dashed arrow marks a negative weight suppressing a contextually wrong sense.
Model
Params
PPL ↓
tax
Transformer
70.1M
21.09
—
DIFF Transformer
70.1M
21.00
−0.09
softmax Backpack
69.6M
24.60
+3.51
differential Backpack (ours)
69.6M
23.52
+2.43
+ DIFF trunk
69.6M
24.78
+3.69
Table 1: WikiText-103 exact validation perplexity at matched parameters and identical training budget (20k steps, single seed); tax is the gap to the plain Transformer.
Figure 2: Interpretability tax on WikiText-103: exact validation perplexity at matched parameters ( ∼ 70M), with a Transformer of the Backpack trunk’s depth ( 8 layers, 51 M) for reference.
Combination rule
PPL ↓
tax
attr. resid.
Transformer
21.09
—
n/a
DIFF Transformer
21.00
−0.09
n/a
Transformer, 8 layers
23.38
+2.29
n/a
one-hot
31.58
+10.49
2⋅10−7
softmax
24.60 ± .12
+3.51
5⋅10−7
diff Backpack (ours)
23.52 ± .04
+2.43
6⋅10−7
Table 2: The constraint ladder on WikiText-103. All rows share the main configuration and budget; tax is the gap to the 14 -layer Transformer; the 8 -layer Transformer ( 51 M) matches the Backpack trunk’s depth; attr. resid. is the relative error of the per-sense decomposition against the model’s own logits.
Sense weights
PPL ↓
% of gain
neg. mass
softmax
24.60
—
0.00
two maps, added
24.30
28%
0.00
scaled softmax
24.51
9%
0.00
λ=0
24.55
5%
0.00
Cog Attention
24.72
−11%
0.39
diff Backpack (ours)
23.52
100%
0.45
Table 3: Controls for the causal claim (WikiText-103, main configuration). “% of gain” is the share of diff’s 1.08 ppl improvement over softmax each rule recovers.
Representation (70M)
POS
STS
SST-2
ARI
Transformer ( h )
0.859
0.488
0.753
0.174
DIFF Transformer ( h )
0.859
0.499
0.743
0.203
softmax Backpack ( o )
0.844
0.372
0.718
0.151
diff Backpack ( o )
0.815
0.495
0.731
0.174
Cog Backpack ( o )
0.836
0.422
0.736
0.163
diff Backpack + MLP ( o )
0.838
0.485
0.749
0.168
Table 4: Frozen-representation probes at 70 M: POS accuracy (UD English-EWT), STS-B Spearman, SST-2 probe accuracy, and sense-clustering ARI, on the Transformer state h or the sense output o ; all scales and MRPC in Appendix D .
Figure 3: Signed self-weight Appℓ of the finance and river senses of bank in a river vs. a financial context, for the differential (left) and softmax (right) Backpacks.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Per-sense negative-mass fraction of the differential Backpack in the main configuration ( k=16 ); 0 is purely excitatory, 1 purely inhibitory.
Rule
ρ
rel. err.
compr.
suffic.
one-hot
1.000
10−6
5.97
+0.27
softmax
1.000
10−6
3.49
−1.43
diff Backpack (ours)
1.000
10−6
3.74
−1.22
unconstr. weights
1.000
10−6
1.14
−4.88
Cog Attention
1.000
10−6
3.63
−1.39
softmax + MLP
0.77
0.48
3.39
−1.45
Appendix
Table 5: Counterfactual faithfulness of the per-sense attribution on WikiText-103 ( 300 positions): Spearman correlation and median relative error between attributed and actual logit change under single removals of the ten top pairs, and the change in target log-probability (nats) when those pairs are removed jointly (comprehensiveness) or kept alone (sufficiency).
Figure 5: Per-token loss increase when inhibition is removed at inference from a trained differential Backpack, by WordNet polysemy of the preceding whole word, for the two edits of § 4.8 ; bars are means with 95% intervals over all validation tokens.
neg.
entropy
sparsity
SL999/WS353
mass
of ∣A∣
@1e-3
(best ρ )
softmax
0.00
0.70
0.66
0.42 / 0.70
diff
0.45
0.78
0.49
0.44 / 0.73
Appendix
Table 6: Sense-weight structure and lexical quality on WikiText-103 (main configuration): negative weight mass, normalized entropy and sparsity of ∣A∣ , and best-sense Spearman ρ on SimLex-999 / WordSim-353 (raw-embedding baselines 0.37 / 0.42 – 0.45 ).
Figure 6: Bias against collateral damage (KL) for the three sense interventions at 70 M, 170 M and 350 M (left to right); upper-left is better.
none
0.5×
remove
suppress
70M
softmax bias
1.14
1.06
0.98
0.86
KL
—
.002
.007
.030
diff bias
1.77
1.65
1.58
1.47
KL
—
.006
.024
.095
170M
softmax bias
1.74
1.42
1.20
1.19
KL
—
.015
.052
.173
Appendix
Table 7: Gender bias ( ↓ ) and collateral damage KL ( ↓ ) under each sense intervention, over 20 profession nouns, at 70 M, 170 M and 350 M.
Scale
Transf.
softmax
diff
gap
cut
70M WT103
21.09
24.60
23.52
1.08 (4.4%)
31%
170M OWT
—
23.84
23.15
0.70 (2.9%)
—
350M OWT
18.02
20.48
19.96
0.52 (2.5%)
21%
Appendix
Table 8: Exact validation perplexity across scale at equal parameters and token budget; gap is softmax minus diff and cut the reduction of the tax (the gap to the parameter-matched Transformer). The 70M and 170M Backpack entries are three-seed means (s.d. 0.12 / 0.04 and 0.13 / 0.03 for softmax/diff); the rest are single runs.
Repr.
POS
STS
SST-2
MRPC
ARI
Transf. ( h ), 70M
0.859
0.488
0.753
0.662
0.174
Transf. 8L ( h ), 70M
0.860
0.484
0.744
0.674
0.160
DIFF Transf. ( h ), 70M
0.859
0.499
0.743
0.654
0.203
softmax ( o ), 70M
0.844
0.372
0.718
0.625
0.151
diff ( o ), 70M
0.815
0.495
0.731
0.613
0.174
Cog ( o ), 70M
0.836
0.422
0.736
0.632
0.163
Appendix
Table 9: Frozen-representation probes at every scale: POS accuracy, STS-B Spearman, SST-2 and MRPC probe accuracy (majority class 0.509 and 0.684 ), and sense-clustering ARI.
Model
best ρ
mean ρ
# ρ>0.6
emb ρ
softmax, k=16
0.70
0.67
6/16
0.45
diff, k=16
0.73
0.69
9/16
0.42
softmax, k=64
0.72
0.58
8/64
0.48
diff, k=64
0.74
0.68
37/64
0.48
Appendix
Table 10: Sense specialization on WordSim-353 (Spearman ρ ): the single best sense, the sense-averaged cosine, the number of individual senses with ρ>0.6 , and the raw token-embedding baseline.
rock , differential Backpack (ours)
mineral
rock, Rock, Rocks, canyon, Canyon
music
rock, garage, punk, gig, pop
rock , softmax Backpack
blended
rock, punk, metal, shale
blended
rock, bedrock, punk, Stones
Appendix
Table 11: Sense lens for rock : the vocabulary items each selected sense most strongly predicts through the tied embedding, C(w)ℓE⊤ , in the two models at k=16 .
Figure 7: Final perplexity by number of senses k on WikiText-103 ( 8 k steps).
squared error
k=1
k=2
k=4
k=8
k=16
softmax
.275
.275
.092
.028
.002
diff
.000
.000
.000
.000
.005
Appendix
Table 12: Extrapolation squared error on the synthetic task of Appendix G by number of sense heads k (mean of 3 seeds).
Figure 8: Mean learned λ over training (a) from three initializations and (b) for three sense counts k (WikiText-103, 8 k steps).
Senses
k
SL999
SV3.5k
RG65
WS353
induced, Transf.
112
0.322
0.179
0.493
0.475
induced, trunk
64
0.365
0.249
0.559
0.552
learned, diff
16
0.441
0.337
0.726
0.731
Appendix
Table 13: Best-sense Spearman ρ of training-free senses induced from OV circuits versus learned Backpack senses.