Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

Maxim Khovansky; Lev Mukhanov

cs.CLOct 24, 2026

Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

Authors: Maxim Khovansky, Lev Mukhanov

Organizations: EECS, Queen Mary University of London, London, United Kingdom

Abstract

Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.

Figures & tables

Name	Role
$\alpha$	Exponent on redundancy $\mathrm{R}_{\ell,h}$ in ( 7 )
$\beta$	Exponent on CLS focus $\mathrm{CF}_{\ell,h}$ in ( 7 )
$\gamma$	Exponent on $\mathrm{Base}$ in ( 6 )
$\delta$	Exponent on $\mathrm{ISF}$ in ( 6 )
$\zeta$	Length normalisation exponent in ( 7 )
$\kappa$	Inverse-weighting scale in ( 8 )

TABLE I: Hyperparameters.

Panel of Figure 2 — Fig. 1: High-level design of SCAPE-T with BERT. [a] The post-aggregation transform $1+\kappa/(\cdot+\epsilon)$ is applied to obtain $\mathrm{ISF}$ ; see ( 8 ).

Metric	WLC Test	Test	Abstractness
SCAPE-T	78.5	85.1	82.3
Lexicon complexity	65.8	94.3	74.4
Flesch–Kincaid	59.5	93.1	78.9
Type–Token ratio	56.4	61.7	50.0
MDL (bits/byte)	52.0	34.1	26.7
Zipf frequency	47.9	85.4	54.6

TABLE II: Pairwise accuracy (%).

Ablation	Pairwise Acc. (%)	$\boldsymbol{\Delta}$ from base (pp)
Baseline	88.80	$0.00$
No head redundancy ( $\alpha{=}0$ )	88.76	$-0.04$
No self-focus penalty ( $\delta{=}0$ )	84.20	$-4.60$
Uniform layer weights ( $\eta{=}0$ )	58.62	$-30.18$
No CLS focus ( $\beta{=}0$ )	57.42	$-31.38$

TABLE III: Submetric ablations on development set.

Ablation	Pairwise Acc. (%)	$\boldsymbol{\Delta}$ from base (pp)
Baseline	82.32	$0.00$
No self-focus penalty ( $\delta{=}0$ )	81.04	$-1.29$
No head redundancy ( $\alpha{=}0$ )	80.25	$-2.08$
Uniform layer weights ( $\eta{=}0$ )	76.29	$-6.04$
No CLS focus ( $\beta{=}0$ )	64.60	$-17.72$

TABLE IV: Submetric ablations on abstractness set.

Panel of Figure 6 — Fig. 2: Per-layer single-removal accuracy drop on development ( $\Delta$ accuracy in pp). Baseline accuracy: 88.8%.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Abbrev.	Meaning
NLP	Natural Language Processing
[CLS]	BERT classification token
[SEP]	BERT separator token
LLM	Large Language Model
MDL	Minimum Description Length
LSA	Latent Semantic Analysis

Appendix

TABLE V: Abbreviations, technical acronyms, and shorthand.

Parameter	Search range
$\alpha$	$[$ 0.0 $,$ 3.0 $]$
$\beta$	$[$ 0.0 $,$ 3.0 $]$
$\gamma$	$[$ 0.0 $,$ 3.0 $]$
$\delta$	$[$ 0.0 $,$ 3.0 $]$
$\zeta$	$[$ 0.0 $,$ 1.0 $]$
$\kappa$	$[$ 0.1 $,$ 5.0 $]$

Appendix

TABLE VI: Hyperparameter bounds.

Jul 20, 2026

Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence

Within Explainable Artificial Intelligence, mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features from neural representations. However, assessing their monosemanticity, and thus explanation quality, remains challenging. Existing metrics require external concept labels or depend on pretrained embedding models, making them sensitive to encoder's geometry. We introduce the Tversky Monosemanticity Score (TMS), a label-free metric that operationalizes monosemanticity as activation-set coherence of binarized SAE latents, and does not require external embedding encoders. We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (DINOv3, CLIP, BLIP2), two common SAE regimes (TopK, BatchTopK), multiple sparsity levels, and expansion factors. Our results show that TMS is less affected by encoder anisotropy than its embedding-based alternative, while remaining aligned with established monosemanticity indicators. TMS also reveals distinct SAE training dynamics across base models. Moreover, under encoder anisotropy, TMS provides a stronger indication of probe-based concept deletion effectiveness, while being competitive otherwise.

Katarzyna Filus, Sebastian Pokuciński

Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland · Department of Applied Informatics, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland

Representation Disentanglement Sparse Autoencoders Mechanistic Interpretability

Ask PDF

Sep 9, 2026

Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties. This matters because readability labels are subjective and rater-dependent, so high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty. We test whether transformers internalize the same features as traditional models across Arabic, English, French, Hindi, and Russian using the ReadMe++ dataset. Shapley Additive Explanations (SHAP) identify the features driving traditional classifiers, which we then use as TCAV concept sets to probe multilingual XLM-R and language-specific encoders. Transformers recover surface-length, syntactic, and lexical-diversity signals, and reflect the ordinal CEFR structure of the traditional models. Alignment varies by model family, language, and layer, with language-specific encoders tracking traditional models more clearly than XLM-R. High linear separability does not always imply directional influence, limiting linear probing for count-based readability features.

Joshua Wong, Chris Tanner

Harvard University · Massachusetts Institute of Technology · Kensho Technologies in Cambridge, MA, USA

Transformer Interpretability Feature Attribution

Ask PDF

Jun 11, 2026

SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection

Prompt-based LLMs are increasingly used for stance detection, but harder examples are not always repaired by clearer instructions, reasoning prompts, retrieval, or debate. We introduce SICI (Stance Inference Complexity Index), a seven-dimensional diagnostic measure of the semantic-pragmatic burden imposed by a target--text pair. Across SemEval-2016 and VAST, SICI predicts LLM accuracy better than surface proxies and shows substantial cross-scorer reliability (

α=0.771

). More importantly, LLM errors change regime as SICI increases: low-complexity examples invite over-attribution, especially Against predictions; intermediate examples form an unstable boundary; and high-complexity examples rapidly concentrate on None. This phase-transition-like structure persists across GPT-3.5, GPT-4o-mini, DeepSeek-V3, and GPT-4o, although stronger models move the boundaries. A 15-method intervention study further shows that prompting, retrieval, and debate often shift models along the attribution--abstention axis rather than removing the high-complexity bottleneck.

Fuqiang Niu, Bowen Zhang

School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China

LLM Evaluation Stance Detection

Ask PDF

Abstract

Explore similar work

Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence

Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection