Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models
Organizations: Center for Electric Mobility Research (CEMOBE) / Power Electronics Laboratories (LEPO), University of Campinas (Unicamp) · Institute of Computing (IC), University of Campinas (Unicamp) · Center for Logic, Epistemology and History of Science (CLE), University of Campinas (Unicamp)
Abstract
A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.
Figures & tables
| Model | Architecture family | Context direction | Reference |
|---|---|---|---|
| bert-base-uncased | Encoder | Bidirectional | ( Devlin et al., 2019 ) |
| roberta-base | Encoder | Bidirectional | ( Liu et al., 2019 ) |
| distilbert-base-uncased | Encoder (distilled) | Bidirectional | ( Sanh et al., 2019 ) |
| microsoft/deberta-base | Encoder | Bidirectional | ( He et al., 2021 ) |
| gpt2 | Decoder | Causal | ( Radford et al., 2019 ) |
| EleutherAI/pythia-410m | Decoder | Causal | ( Biderman et al., 2023 ) |
Representation Geometry in Language Models
Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.
See all 421 papers