Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

Maxim Khovansky; Lev Mukhanov

cs.CLOct 24, 2026

Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

Authors: Maxim Khovansky, Lev Mukhanov

Organizations: EECS, Queen Mary University of London, London, United Kingdom

Abstract

Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.

Figures & tables

Name	Role
$\alpha$	Exponent on redundancy $\mathrm{R}_{\ell,h}$ in ( 7 )
$\beta$	Exponent on CLS focus $\mathrm{CF}_{\ell,h}$ in ( 7 )
$\gamma$	Exponent on $\mathrm{Base}$ in ( 6 )
$\delta$	Exponent on $\mathrm{ISF}$ in ( 6 )
$\zeta$	Length normalisation exponent in ( 7 )
$\kappa$	Inverse-weighting scale in ( 8 )

TABLE I: Hyperparameters.

Panel of Figure 2 — Fig. 1: High-level design of SCAPE-T with BERT. [a] The post-aggregation transform $1+\kappa/(\cdot+\epsilon)$ is applied to obtain $\mathrm{ISF}$ ; see ( 8 ).

Metric	WLC Test	Test	Abstractness
SCAPE-T	78.5	85.1	82.3
Lexicon complexity	65.8	94.3	74.4
Flesch–Kincaid	59.5	93.1	78.9
Type–Token ratio	56.4	61.7	50.0
MDL (bits/byte)	52.0	34.1	26.7
Zipf frequency	47.9	85.4	54.6

TABLE II: Pairwise accuracy (%).

Ablation	Pairwise Acc. (%)	$\boldsymbol{\Delta}$ from base (pp)
Baseline	88.80	$0.00$
No head redundancy ( $\alpha{=}0$ )	88.76	$-0.04$
No self-focus penalty ( $\delta{=}0$ )	84.20	$-4.60$
Uniform layer weights ( $\eta{=}0$ )	58.62	$-30.18$
No CLS focus ( $\beta{=}0$ )	57.42	$-31.38$

TABLE III: Submetric ablations on development set.

Ablation	Pairwise Acc. (%)	$\boldsymbol{\Delta}$ from base (pp)
Baseline	82.32	$0.00$
No self-focus penalty ( $\delta{=}0$ )	81.04	$-1.29$
No head redundancy ( $\alpha{=}0$ )	80.25	$-2.08$
Uniform layer weights ( $\eta{=}0$ )	76.29	$-6.04$
No CLS focus ( $\beta{=}0$ )	64.60	$-17.72$

TABLE IV: Submetric ablations on abstractness set.

Panel of Figure 6 — Fig. 2: Per-layer single-removal accuracy drop on development ( $\Delta$ accuracy in pp). Baseline accuracy: 88.8%.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Abbrev.	Meaning
NLP	Natural Language Processing
[CLS]	BERT classification token
[SEP]	BERT separator token
LLM	Large Language Model
MDL	Minimum Description Length
LSA	Latent Semantic Analysis

Appendix

TABLE V: Abbreviations, technical acronyms, and shorthand.

Parameter	Search range
$\alpha$	$[$ 0.0 $,$ 3.0 $]$
$\beta$	$[$ 0.0 $,$ 3.0 $]$
$\gamma$	$[$ 0.0 $,$ 3.0 $]$
$\delta$	$[$ 0.0 $,$ 3.0 $]$
$\zeta$	$[$ 0.0 $,$ 1.0 $]$
$\kappa$	$[$ 0.1 $,$ 5.0 $]$

Appendix

TABLE VI: Hyperparameter bounds.

Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence
Jul 20, 2026Katarzyna Filus, Sebastian PokucińskiRepresentation Disentanglement Sparse Autoencoders
Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland · Department of Applied Informatics, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland
Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment
Sep 9, 2026Joshua Wong, Chris TannerTransformer Interpretability Feature Attribution
Harvard University · Massachusetts Institute of Technology · Kensho Technologies in Cambridge, MA, USA
SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection
Jun 11, 2026Fuqiang Niu, Bowen ZhangLLM Evaluation Stance Detection
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China

Jul 20, 2026J/K move · Enter open · S save

Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence

Katarzyna Filus, Sebastian Pokuciński

Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland · Department of Applied Informatics, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland

Representation Disentanglement Sparse Autoencoders Mechanistic Interpretability

Open paper AskPDF arXiv

Within Explainable Artificial Intelligence, mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features from neural representations. However, assessing their monosemanticity, and thus explanation quality, remains challenging. Existing metrics require external concept labels or depend on pretrained embedding models, making them sensitive to encoder's geometry. We introduce the Tversky Monosemanticity Score (TMS), a label-free metric that operationalizes monosemanticity as activation-set coherence of binarized SAE latents, and does not require external embedding encoders. We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (DINOv3, CLIP, BLIP2), two common SAE regimes (TopK, BatchTopK), multiple sparsity levels, and expansion factors. Our results show that TMS is less affected by encoder anisotropy than its embedding-based alternative, while remaining aligned with established monosemanticity indicators. TMS also reveals distinct SAE training dynamics across base models. Moreover, under encoder anisotropy, TMS provides a stronger indication of probe-based concept deletion effectiveness, while being competitive otherwise.

Abstract

Explore similar work

Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence

Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection