From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory

Jason Dury

cs.AIMar 19, 2026

From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory

Authors: Jason Dury

Organizations: Independent Researcher

Abstract

Embedding models group text by semantic content, what text is about. We show that temporal co-occurrence within texts discovers a different kind of structure: recurrent transition-structure concepts or what text does. We train a 29.4M-parameter contrastive model on 373 million co-occurrence pairs from 9,766 Project Gutenberg texts (24.96 million passages), mapping pre-trained embeddings into an association space where passages with similar transition structure cluster together. Under capacity constraint (42.75% accuracy), the model must compress across recurring patterns rather than memorise individual co-occurrences. Clustering at six granularities (k=50 to k=2,000) produces a multi-resolution concept map; from broad modes like "direct confrontation" and "lyrical meditation" to precise registers and scene templates like "sailor dialect" and "courtroom cross-examination." At k=100, clusters average 4,508 books each (of 9,766), confirming corpus-wide patterns. Direct comparison with embedding-similarity clustering shows that raw embeddings group by topic while association-space clusters group by function, register, and literary tradition. Unseen novels are assigned to existing clusters without retraining; the association model concentrates each novel into a selective subset of coherent clusters, while raw embedding assignment saturates nearly all clusters. Validation controls address positional, length, and book-concentration confounds. The method extends Predictive Associative Memory (PAM, arXiv:2602.11322) from episodic recall to concept formation: where PAM recalls specific associations, multi-epoch contrastive training under compression extracts structural patterns that transfer to unseen texts, the same framework producing qualitatively different behaviour in a different regime.

Figures & tables

$k$	Threshold	Passing	Mean Cosine	Mean Books	Mean Dominance
50	50	50/50	0.302	5,860	2.2%
100	100	100/100	0.378	4,508	4.0%
250	200	241/250	0.436	3,329	6.3%
500	500	472/500	0.479	2,502	9.0%
1,000	500	857/1,000	0.513	1,885	12.0%
2,000	1,000	980/2,000	0.499	1,797	8.1%

Table 1: Cluster statistics at each granularity. “Mean Books” is the average number of distinct books per cluster. “Mean Dominance” is the average maximum single-book fraction. “Mean Cosine” is mean intra-cluster cosine similarity in association space.

Figure 1: Method overview. Texts are chunked into short passages, embedded with BGE-large-en-v1.5, and temporal co-occurrence pairs are extracted within a 15-chunk window. A contrastive MLP maps embeddings into association space; $k$ -means clustering at six granularities ( $k{=}50$ to $k{=}2{,}000$ ) produces a multi-resolution concept map.

Metric	PAM 2K ( $k{=}100$ )	BGE 2K ( $k{=}100$ )	PAM 10K ( $k{=}100$ )
Mean cosine	0.454	0.493	0.378
Mean books	1,121	1,905	4,508
Mean dominance	7.9%	2.3%	4.0%

Table 2: Matched 2K comparison. The PAM 10K column is included for cross-scale reference, not as a matched comparison.

Metric ( $k{=}100$ )	BGE Raw	Context-Enriched	PAM 2K
Passing	100/100	95/100	100/100
Mean cosine	0.493	0.861	0.454
Mean books	1,905	726	1,121
Mean dominance	2.3%	8.1%	7.9%

Table 3: Context-enriched baseline vs BGE and PAM on the matched 2K corpus.

Novel	Gutenberg ID	Author	Chunks
Pride and Prejudice	1342	Jane Austen	4,697
Dracula	345	Bram Stoker	5,879
Frankenstein	84	Mary Shelley	2,638
Alice’s Adventures in Wonderland	11	Lewis Carroll	1,057
The War of the Worlds	36	H.G. Wells	2,157

Table 4: Unseen evaluation novels.

Figure 2: BGE similarity-based clusters group passages by topic (what text is about ), while PAM association-based clusters group by transition structure (what text does )—narrative function, discourse register, and literary tradition.

Figure 3: Multi-resolution concept structure from $k{=}50$ to $k{=}2{,}000$ . Broad narrative modes at coarse resolution decompose into increasingly specific variants—registers, traditions, and scene templates—at finer granularity.

Figure 4: Authorial pacing signatures for three canonical works. Different authors produce characteristically different distributions of structural modes, visible at multiple resolutions simultaneously.

Novel	PAM $k{=}100$	BGE $k{=}100$	Top-5 PAM	Top-5 BGE
Alice in Wonderland	51/100	87/100	77.6%	32.2%
Pride and Prejudice	80/100	89/100	66.5%	25.2%
Frankenstein	83/100	96/100	60.6%	42.4%
The War of the Worlds	86/100	86/100	52.6%	36.7%
Dracula	98/100	100/100	39.1%	19.5%

Table 5: Cluster activation and top-5 concentration for unseen novels, ordered by PAM selectivity. “Top-5 PAM/BGE” is the fraction of a novel’s passages assigned to its five most frequent clusters—higher means a more concentrated concept profile.

Figure 5: Cluster selectivity for unseen novels. PAM concentrates each novel into a selective subset of structurally coherent clusters, while BGE assignment saturates nearly all clusters.

Property	PAM	AAR	This Work
Domain	Synthetic benchmark	Multi-hop QA	Literary corpus
Goal	Faithful episodic recall	Passage retrieval	Concept discovery
Compression	None (memorisation)	Minimal ( $\sim$ 97%)	Severe (42.75%)
Inductive transfer	Fails	Fails	Succeeds
What is learned	Specific associations	Corpus-specific links	Structural patterns
Biological analogy	Hippocampal encoding	—	Neocortical consolidation

Table 6: Comparison of PAM, AAR, and this work across key properties.

Unsupervised Visual Concept Discovery

See all 13 papers

Aug 29, 2026

A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes

Recent topic models leverage pretrained embeddings, but neural architectures produce latent representations without grounding in specific texts, and clustering-based pipelines assign representative documents only post hoc, relying on absolute distances distorted by hubness and anisotropy in high-dimensional spaces. We introduce MARETopic, a training-free framework that casts topic discovery as rank-based prototype selection. After projecting embeddings onto a low-dimensional manifold, MARETopic builds ranked lists encoding ordinal neighborhood structure. A greedy algorithm selects exactly K exemplar documents, real corpus texts, whose neighborhoods cover the corpus. Two variants share this criterion. MARETopic

_\text{Corr}

scores candidates with a query performance predictor and a rank correlation measure, leading Purity and NMI on the two benchmarks with the most categories, ahead of both neural and clustering-based topic models. MARETopic

_\text{Diff}

scores them with a rank-based diffusion matrix, needs neither measure, and runs 1.7 to 1.9 times faster. Without a single gradient update, MARETopic leads topic coherence on two of three datasets. A novel inter-topic Maximal Marginal Relevance step raises vocabulary diversity at little cost in coherence. Our code is available at https://github.com/thcastilho/maretopic.

Thiago César Castilho Almeida, Daniel Carlos Guimarães Pedronette

Department of Statistics, Applied Mathematics and Computing State University of São Paulo (UNESP), Rio Claro, Brazil

Topic Modeling

Ask PDF

May 6, 2026

Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement

We introduce the Concept Field of a text corpus: a local drift field with pointwise uncertainty, estimated in sentence-embedding space from the deltas between consecutive sentences. Given a candidate sentence transition, we score its agreement with the field by

ζ

, the mean absolute z-distance between the observed delta and the field's local Gaussian estimate. The score is black-box (no model internals), corpus-attributable (every score traces to nearby corpus sentences), and admits a probabilistically motivated interpretation under a local Gaussian approximation. We support the computation with the introduction of a Vector Sequence Database (VSDB) that stores embeddings together with sequence-position and next-delta metadata. We evaluate this approach on two large-scale settings: hallucination-style groundedness detection over the U.S. Code of Federal Regulations, and novelty detection over Project Gutenberg. On controlled LLM-generated rewrites, Concept Fields achieve strong selective classification performance under a grounded / ungrounded / unsure triage policy. Unlike retrieval-centric baselines, the resulting coverage-risk behavior is similar across both domains, supporting a degree of cross-domain stability for the standardized deviation score. We also sketch how divergence and curl of the Concept Field, computed on dense clusters, surface qualitatively meaningful semantic patterns (logic sources, sinks, and implicit topics), which we offer as hypothesis-generating rather than as a quantitative result. Concept Fields provide a fast, lightweight, and interpretable signal for groundedness and novelty, complementary to LLM-as-judge and white-box detectors.

Nicholas S. Kersting, Vittorio Castelli, Chieh Ting Yeh +2

Oracle Corporation

LLM Grounding LLM Hallucination Detection

Ask PDF

Jul 31, 2025

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic steering vectors. To properly evaluate MTM topics against word list approaches, we propose topic judge, an LLM-based pairwise comparison evaluation framework. Across eight datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering.

Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar +5

Columbia University · Independent · Google Research

Topic Modeling

Ask PDF

Abstract

Unsupervised Visual Concept Discovery

Explore similar work

A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes

Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders