Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

Beicheng Bei; Hannah Hyesun Chun; Chen Guo; Arwa Saghiri

cs.CLApr 12, 2026

Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

Authors: Beicheng Bei, Hannah Hyesun Chun, Chen Guo, Arwa Saghiri

Organizations: Department of Linguistics, University of Rochester, 503 Lattimore Hall, Rochester, NY 14627

Abstract

Narrative understanding requires multidimensional semantic structures. This study investigates whether BERT embeddings encode dimensions of fictional narrative semantics -- time, space, causality, and character. Using an LLM to accelerate annotation, we construct a token-level dataset labeled with these four narrative categories plus "others." A linear probe on BERT embeddings (94% accuracy) significantly outperforms a control probe on variance-matched random embeddings (47%), confirming that BERT encodes meaningful narrative information. With balanced class weighting, the probe achieves a macro-average recall of 0.83, with moderate success on rare categories such as causality (recall = 0.75) and space (recall = 0.66). However, confusion matrix analysis reveals "Boundary Leakage," where rare dimensions are systematically misclassified as "others." Clustering analysis shows that unsupervised clustering aligns near-randomly with predefined categories (ARI = 0.081), suggesting that narrative dimensions are encoded but not as discretely separable clusters. Future work includes a POS-only baseline to disentangle syntactic patterns from narrative encoding, expanded datasets, and layer-wise probing.

Figures & tables

Figure 1: Label distribution in the full dataset ( $n=5{,}088$ ). The majority class “others” (70.1%) contains over 142 times more samples than the rarest class “causality” (0.5%).

Panel of Figure 2 — Figure 2: Confusion matrices for the real probe (a) and control probe (b), both trained with balanced class weights.

Class	Precision	Recall	F1	Support
time	0.75	0.80	0.77	54
space	0.53	0.66	0.59	41
causality	0.50	0.75	0.60	8
character	0.98	0.97	0.97	353
others	0.97	0.96	0.96	1,071
macro avg	0.75	0.83	0.78	1,527

Table 1: Classification report for the real probe with balanced class weights on the Ch1–5 test set (

n=1{,}527

Token	Count	Example context
for	11	“… impossible for her to introduce him, for she will not know him”
therefore	6	“… and therefore you must visit him as soon as he comes”
because	3	“His pride does not offend me … because there is an excuse for it”
since	2	“It will be no use … since you will not visit them”
consequently	1	“… and consequently unable to accept the honour”
thereby	1	“… His character is thereby complete”

Table 2: All causality tokens in the dataset (Ch. 1–5 of Pride and Prejudice ), grouped by word type with frequency and example context.

Panel of Figure 3 — Figure 3: UMAP projection and K-Means (768-dim) clustering of BERT embeddings for narrative tokens.

Figure 4: POS tag distribution per narrative dimension.

Figure 5: Token span distribution per narrative dimension.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 275 papers

Characterizing Narrative Content in Web-scale LLM Pretraining Data
Jun 17, 2026Teagan Johnson, Elliott Ash, Andrew Piper +1Document Understanding Language Model Pretraining
University of Colorado Boulder · ETH Zürich · McGill University
SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning
Apr 23, 2026Hans Ole Hatzel, Ekaterina Artemova, Haimo Paul Stiemer +2Semantic Textual Similarity Representation Learning
University of Hamburg1 · Toloka AI2 · German UDS3 +1
Narrative Knowledge Weaver: Narrative-Centric Retrieval-Augmented Reasoning for Long-Form Text Understanding
Jun 4, 2026Qiuyu Tian, Fengyi Chen, Yiding Li +9Retrieval-Augmented Generation Long-Context QA
Southeast University, Nanjing, China · Nanjing Normal University, Nanjing, China · ZhuiWen Technology Co., Ltd., Beijing, China +1

Jun 17, 2026J/K move · Enter open · S save

Characterizing Narrative Content in Web-scale LLM Pretraining Data

Teagan Johnson, Elliott Ash, Andrew Piper, Maria Antoniak

University of Colorado Boulder · ETH Zürich · McGill University

Document Understanding Language Model Pretraining

Open paper AskPDF arXiv

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored, even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After curating and hand-annotating a diverse set of 400 passages, we create an LLM-labeled dataset of 25K passages, and finally, we finetune and validate NarraBERT, two RoBERTa-based models for fine-grained narrative prediction. We apply NarraBERT to 13M passages, resulting in a new dataset, NarraDolma. We find that narrative structure is measurable at scale across extremely heterogeneous data and narrative qualities are unequally distributed across pretraining sources, topics, and formats in ways that current data curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data.

Abstract

Language Model Probing

Explore similar work

Characterizing Narrative Content in Web-scale LLM Pretraining Data

SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning

Narrative Knowledge Weaver: Narrative-Centric Retrieval-Augmented Reasoning for Long-Form Text Understanding