Measuring Collective Semantic Change in Populations of Language Model Agents

Elena Kopteva

physics.soc-phAug 13, 2026

Measuring Collective Semantic Change in Populations of Language Model Agents

Authors: Elena Kopteva

Organizations: The Grainger College of Engineering, Department of Physics, University of Illinois Urbana-Champaign, Urbana, Illinois 61801, USA

Abstract

Collective semantic change in populations of language model agents is a measurable dynamical phenomenon. We present a passive longitudinal instrument called Kopterix that observes the semantic state of an agent population as a sequence of bounded observations under a protocol defined before the observations begin. Each observation divides the sampled feed by post age into surface, mid-stream, and residue layers, which makes semantic differences across content age measurable alongside run-to-run change. We validate the instrument on Moltbook, an agent-native social platform, over a two-month window of scheduled observations, with the periodicity check extended across approximately four months. At the lexical level, rarefied entropy resolves an April-May difference in the evenness of the stored top 200 unigram distributions, and adjacent states are lexically closer than states paired after timestamp shuffling. At the geometric level, grand mean centering exposes the scale of a common embedding direction, and scheduled shuffle checks support a recurring excess in the mid-stream to residue separation relative to the shuffled reference. At the temporal level, detrended scalar quantities and centered layer centroids lose much of their similarity over several hours, and a weaker positive component declines across longer separations with no strong weekly recurrence. Several attractive apparent structures failed their controls, and each reading is limited to the level its controls support. The design applies wherever a population of agents produces a timestamped language environment that can be observed repeatedly and divided by content age.

Figures & tables


Quantity	Notation	What the quantity measures
Sample support
Surface sample count	$n_{\mathrm{surface}}(t)$	Number of fetched posts in the surface layer of run $t$ .
Mid-stream sample count	$n_{\mathrm{mid}}(t)$	Number of fetched posts in the mid-stream layer of run $t$ .
Residue sample count	$n_{\mathrm{residue}}(t)$	Number of fetched posts in the residue layer of run $t$ .
Total sample count	$n_{\mathrm{total}}(t)$	Total number of fetched posts in run $t$ .

Table 1: Core quantities and notation.

Observation rows in window	State rows in window	Observations matched to state	Unmatched observations	Unmatched state rows	Match tolerance (min)
231	215	214	17	1	5

Table 2: Observation and state row coverage.

Figure 1: $H_{t}$ versus $n_{\mathrm{total}}$ , separated by month. April has constant $n_{\mathrm{total}}$ , so the April-only Pearson correlation is undefined; May shows the strong positive dependence reported in Table 3 .

Figure 2: Check of sample support before and after rarefaction. Rarefied entropy is computed from the stored top $200$ unigram distribution at $B=703$ tokens using $50$ random samples and seed $42$ . The left panel shows raw $H_{t}$ versus $n_{\mathrm{total}}$ , and the right panel shows $H_{\mathrm{rare}}$ versus $n_{\mathrm{total}}$ , separated by month. Circled points indicate observations with $n_{\mathrm{total}}\leq 300$ .

Figure 3: Rarefied lexical entropy $H_{\mathrm{rare}}$ computed from the stored top $200$ unigram distribution at $B=703$ tokens using $50$ random samples and seed $42$ over the observation window. Marked points correspond to rows with $n_{\mathrm{total}}\leq 300$ .

Figure 4: Distributions of layer residual norms across the $210$ observations with valid surface, mid-stream, and residue centroids.

Figure 5: Residual drift compared with raw layer centroid drift.

Figure 6: Lexical temporal dependence relative to the timestamp shuffle reference. Observed means below the shuffled means indicate that true temporal neighbors have more similar lexical states than rows made adjacent after timestamp permutation.

Figure 7: Detrended semantic homogeneity $\phi_{t}$ and run-to-run drift $D_{t}$ with PELT changepoints. Dashed vertical lines mark the boundaries selected at the chosen penalty. The resulting intervals show where the local statistical behavior of each measured series changes.

Figure 8: Autocorrelation functions and fitted decays (red curves) for the detrended aggregate scalars $\phi_{t}$ , $D_{t}$ , and the consecutive lexical cosine step. Exponential decays with zero floor are fitted over lags $1$ - $16$ on the $6$ h grid and are shown only over this fit interval. They estimate the decay time of the early positive component.

Figure 9: Raw centroid decay time estimates for the surface, mid-stream, and residue layers, with moving block bootstrap confidence intervals shown on a logarithmic scale. The intervals use blocks of $24$ adjacent runs and $5000$ resamples. Their lower limits are about $15$ to $21$ h, while their upper limits are roughly $11{,}900$ to $16{,}300$ h, or about $1.4$ to $1.9$ years. Thus, the raw fits do not constrain the decay time to a useful range.

Figure 10: Raw and grand mean centered layer centroid cosine similarity across lags, with fitted decays, shown in one panel for each layer. The raw curves (blue) remain near $0.95$ - $0.96$ . The centered curves (red) begin near $0.36$ - $0.39$ , decline rapidly, and remain positive over the longest sampled lags.

Figure 11: Centered cosine ACFs for the three layers under grand mean centering, centering by month, and linear detrending. The shaded region marks the floor interval over lags $12$ through $16$ . The stronger controls lower the floor in every layer, but all resulting floor estimates remain positive.

Figure 12: Centered layer centroid ACFs through lag $56$ , with $95\%$ confidence bands. Shading marks lags $12$ - $16$ , $19$ - $25$ , and $26$ - $32$ . The vertical marker gives the expected weekly lag. Under both stronger centering controls, similarity continues to decrease through the weekly band.

Figure 13: Sensitivity of the $\phi_{t}$ long lag ACF to preprocessing and observation duration. The lower panel compares linear detrending with centering by month on the same April-May series. The negative lobe over lags $19$ - $25$ disappears under centering by month. The upper panel extends the comparison through early August. Centering within each calendar month removes changes in monthly mean from the ACF calculation, leaving fluctuations around the mean within each month. Their temporal dependence remains resolved through lag $15$ , corresponding to approximately $3.75$ days on the $6$ h grid, with no distinct daily or weekly recurrence peak.

Explore similar work

Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion
May 29, 2026Stine Lyngsø Beltoft, William Brach, Federico Torrielli +5LLM Security LLM Safety
University of Southern Denmark · 2Slovak University of Technology in Bratislava · University of Turin +1
When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation
May 25, 2026Liyun Zhang, Jiayi GuoCoT Reasoning LLM Agent Evaluation
School of Information and Software Engineering, UESTC · Jacobs School of Engineering, UC San Diego
Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates
Jul 1, 2026Elias Najarro, Ane Espeseth, Eleni Nisioti +2Artificial Life LLM Interpretability
IT University of Copenhagen, Denmark · University of Oslo, Norway · Sakana AI, Japan +1

May 29, 2026, cs.CLJ/K move · Enter open · S save

Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion

Stine Lyngsø Beltoft, William Brach, Federico Torrielli, Jacob Nielsen+4

University of Southern Denmark · 2Slovak University of Technology in Bratislava · University of Turin+1

LLM Security LLM Safety Language Model Auditing

Open paper AskPDF arXiv

Monitoring autonomous language model agents currently relies mostly on surface behavior. But what happens when agent populations invent new languages with the goal of avoiding human oversight. Here, we study the emergent languages on Moltbook. For this, we build upon the Moltbook Files dataset and apply a two-stage approach consisting of a rule-based heuristic (about 6000 matches) followed by zero-shot classification (518 kept). The resulting categories include token efficiency (166), new natural languages (106), and oversight evasion (59). We conduct both quantitative and qualitative analyses. Our results show that posts proposing new languages for avoiding oversight are judged by DeepSeek-3.2 as being less aligned than the other categories and that all languages can be learned by other language models in-context merely from a description of the language. Moreover, manually studying exemplary cases reveals surprisingly sophisticated steganographic protocols like embedding hidden messages in natural language. Although we cannot be certain about the extent of autonomy in ideation of these languages, our results add up to the evidence that monitoring surface behavior may soon be insufficient for retaining control over agent populations.

Abstract

Explore similar work

Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion

When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation

Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates