Factorized Representations of Concurrent Temporal Variables in Pretrained Language Models

Xinhe Zhang; Arnau Marin-Llobet

cs.LGOct 29, 2026

Factorized Representations of Concurrent Temporal Variables in Pretrained Language Models

Authors: Xinhe Zhang, Arnau Marin-Llobet

Abstract

Language models often need to maintain multiple contextual variables that evolve independently within a document. Prior work has identified structured representations of individual temporal or contextual quantities, but their joint organization remains less understood. We study this question using controlled documents with independently phased language-switch and topic-change timers. Across ten pretrained checkpoints, both timers are linearly decodable from the same hidden states. Removing a timer-associated centroid plane substantially reduces access to its associated variable while having little effect on the concurrent timer, although the planes are nonorthogonal and contain cross-timer information. Separate timer contributions also explain most reproducible variation among joint-state centroids. These signatures persist from 3 to 51 timer values and across Qwen3 checkpoints from 0.6B to 14B. Schedule position affects continuation probabilities, but the tested centroid displacement produces no consistent timer-specific output effect. Together, these results support approximate factorization based on selective accessibility and approximately additive joint-state geometry in overlapping subspaces.

Figures & tables

Figure 1: Selective ablation of concurrent timer representations. A , Independently phased language and topic schedules. B–C , Cross-fitted decoding after subspace removal and decoder retraining. Points represent models; bars indicate means.

Figure 2: Information overlap and approximately additive joint geometry. A–B , Decoding restricted to timer-associated planes. C , Joint-centroid interaction residuals across models. Pale bars indicate the mean

\pm

bootstrap standard error.

$H$	Values	$\rho$	Rank	Self	Cross	Random	Cross/self
2	3	0.91	1.75	84.9%	0.15%	0.18%	0.18%
5	6	0.89	2.50	75.3%	0.44%	0.31%	0.56%
10	11	0.87	3.25	79.9%	2.05%	0.53%	2.58%
20	21	0.87	3.48	79.8%	2.96%	0.69%	3.71%
50	51	0.86	3.98	84.6%	4.86%	0.76%	5.71%

Table 1: Cross-fitted horizon sweep across four models. Rank is the mean selected dimension; Self, Cross, and Random are percentage reductions in intact Spearman correlation. See Appendix D.3 for aggregation details.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Sent.	Text	Lang.	Topic	$y_{\mathrm{lang}}$	$y_{\mathrm{topic}}$
0	The lamp is bright.	EN	house	1	3 †
1	The mountain is high.	EN	field	2	0
2	Der Fluss ist breit.	DE	field	0	1
3	Der Stein ist grau.	DE	field	1	2
4	Der Wald ist dunkel.	DE	field	2	3 †
5	The chair is heavy.	EN	house	0	0

Appendix

Table 2: Excerpt from a family-A document with language and topic timer labels. † Topic-phase-3 rows are excluded from primary three-class topic analyses.

Paper name	Checkpoint	Layers	Analyzed
Llama-3.2-1B	meta-llama/Llama-3.2-1B	16	8
Mistral-7B-v0.3	mistralai/Mistral-7B-v0.3	32	16
Qwen3-1.7B	Qwen/Qwen3-1.7B	28	14
Qwen3-1.7B-Base	Qwen/Qwen3-1.7B-Base	28	14
Qwen3-4B	Qwen/Qwen3-4B	36	18
pythia-1.4b	EleutherAI/pythia-1.4b	24	12

Appendix

Table 3: Checkpoints, transformer-block counts, and analyzed layer indices.

	Language timer			Topic timer
Model	Intact	Own	Other	Intact	Own	Other
Llama-3.2-1B	0.891	0.423	0.886	0.460	0.390	0.459
Mistral-7B	0.917	0.433	0.909	0.513	0.403	0.506
Qwen3-1.7B	0.878	0.421	0.867	0.502	0.404	0.498
Qwen3-1.7B-Base	0.941	0.431	0.938	0.531	0.412	0.528
Qwen3-4B	0.864	0.428	0.851	0.522	0.414	0.512

Appendix

Table 4: Cross-fitted balanced accuracy before ablation (Intact), after own-plane removal (Own), and after concurrent-timer-plane removal (Other).

	Own, random	Own, phase-matched	Own, within-document	Other, random
Language	84.5%	84.0%	81.4%	1.6%
Topic	58.4%	63.0%	53.6%	2.3%

Appendix

Table 5: Percentage of above-chance decoding signal removed under each split condition, averaged over per-model removal fractions from a separate eight-draw experiment. Own and Other denote removal of the target timer’s plane and the concurrent timer’s plane, respectively.

Timer	Intact	Own, cross-fitted	Other, cross-fitted	Own removed	Other removed
Language	0.893	0.576	0.886	81%	1.7%
Topic	0.598	0.553	0.591	45%	6.3%

Appendix

Table 6: Balanced accuracy and percentage of above-chance signal removed in history-only decoding. Own and Other denote ablation of the target timer’s plane and the concurrent timer’s plane, respectively.

Transferred fraction $\alpha$	Cross-ablation	Self-ablation	Above-chance accuracy
0.00	0.6	16.5	+0.164
0.25	1.1	11.2	+0.109
0.50	4.4	8.1	+0.081
0.75	10.4	8.5	+0.084
1.00	11.0	10.8	+0.110

Appendix

Table 7: Ablation effects under imposed overlap, reported as balanced-accuracy decreases in percentage points. The final column reports intact balanced accuracy minus chance.

Model	Own plane	Concurrent plane	Placebo	Angle
SmolLM2-1.7B	+0.596	+0.136	+0.164	$66.6^{\circ}$
Qwen3-1.7B-Base	+0.537	+0.259	+0.149	$53.4^{\circ}$
OLMo-2-1B	+0.528	+0.188	+0.146	$52.4^{\circ}$
Mistral-7B-v0.3	+0.515	+0.206	+0.166	$53.1^{\circ}$
Llama-3.2-1B	+0.451	+0.153	+0.117	$55.4^{\circ}$
gemma-2-2b	+0.438	+0.217	+0.105	$41.6^{\circ}$

Appendix

Table 8: Above-chance language-timer decoding from the language plane (Own), topic plane (Concurrent), and placebo plane, with the smaller principal angle between the two timer planes.

	Interaction residual		Held-out variance explained
Model	Observed	Additive floor	Additive	Saturated
Llama-3.2-1B	9.1%	4.3%	0.839	0.833
Mistral-7B	6.8%	3.4%	0.876	0.852
Qwen3-1.7B	12.8%	6.0%	0.758	0.761
Qwen3-1.7B-Base	7.5%	3.1%	0.868	0.874
Qwen3-4B	12.8%	6.1%	0.767	0.753

Appendix

Table 9: Joint-centroid interaction residuals and held-out variance explained by additive and saturated nine-centroid models. The additive floor estimates the finite-sample residual under exactly additive population means.

Timer	Across	Split-half	Across / split-half	Shuffled
Language	0.855	0.889	96%	+0.005
Topic	0.331	0.587	54%	+0.005

Appendix

Table 10: Mean cosine similarity of timer-transition displacements across concurrent-timer values (Across), between document halves at a fixed concurrent-timer value (Split-half), and under shuffled labels. Across / split-half reports the mean of per-model ratios.

Figure 13

Model	Interaction $\pm$ SEM	Shuffled	Split-half angle (lang./topic)
Qwen3-0.6B	$12.1\pm 1.7\%$	50.4%	$17.2^{\circ}/58.7^{\circ}$
Qwen3-1.7B	$14.6\pm 2.1\%$	50.6%	$22.2^{\circ}/51.7^{\circ}$
Qwen3-4B	$15.9\pm 2.4\%$	49.9%	$27.0^{\circ}/51.7^{\circ}$
Qwen3-8B	$13.3\pm 2.1\%$	50.1%	$19.7^{\circ}/53.4^{\circ}$
Qwen3-14B	$12.7\pm 2.0\%$	50.3%	$20.2^{\circ}/41.1^{\circ}$

Appendix

Table 11: Interaction residuals and plane reproducibility across Qwen3 model sizes. Uncertainty is one bootstrap standard error estimated from 40 document resamples; Shuffled reports the shuffled-label interaction residual. Shuffled-label plane angles range from

84^{\circ}

86^{\circ}

Model	$A_{\mathrm{due}}$	$A_{\mathrm{control}}$	$D\pm$ SEM
pythia-1.4b	3.301	-0.837	$4.138\pm 0.658$
Qwen3-1.7B	2.147	-0.741	$2.888\pm 0.556$
Phi-3-mini	2.094	-0.750	$2.844\pm 0.456$
OLMo-2-1B	1.997	-0.767	$2.764\pm 0.569$
Llama-3.2-1B	2.262	-0.280	$2.542\pm 0.418$
SmolLM2-1.7B	1.962	-0.449	$2.411\pm 0.546$

Appendix

Table 12: Recency-matched continuation contrasts. Uncertainty is one standard error from 2000 document-bootstrap resamples. The final row reports a randomly initialized Qwen3-1.7B control.

Model	Target	Concurrent	Random	Target $-$ concurrent
Phi-3-mini	0.0249	0.0231	0.0010	0.0018
gemma-2-2b	0.0243	0.0066	-0.0070	0.0177
OLMo-2-1B	0.0173	0.0499	-0.0015	-0.0326
pythia-1.4b	0.0126	-0.0164	0.0037	0.0290
Mistral-7B-v0.3	0.0046	0.0215	-0.0036	-0.0169
SmolLM2-1.7B	0.0021	0.0324	-0.1014	-0.0303

Appendix

Table 13: Mean continuation log-odds change at strength

1.0

for language-timer (Target), concurrent-timer, and isotropic (Random) interventions.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 421 papers

ChronoState: Hidden Elapsed-Time Conditioning for Temporal-State Action Selection in Frozen-Backbone Language Models
Aug 10, 2026Sam Siavoshian, Omar Ramadan, Amir K. Saeed +3Temporal Reasoning in Language Models Language Modeling
Independent Researcher, California, USA · Johns Hopkins University, Whiting School of Engineering, Baltimore, MD, USA
Evaluating Temporal Consistency in Multi-Turn Language Models
Apr 24, 2026Yash Kumar Atri, Steven L. Johnson, Tom HartvigsenLLM Evaluation Temporal Reasoning in Language Models
University of Virginia
What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?
Mar 19, 2026Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1Tokenization in Language Models Temporal Reasoning in Language Models
University of Aberdeen · Université Grenoble Alpes & CNRS

Aug 10, 2026J/K move · Enter open · S save

ChronoState: Hidden Elapsed-Time Conditioning for Temporal-State Action Selection in Frozen-Backbone Language Models

Sam Siavoshian, Omar Ramadan, Amir K. Saeed, Benjamin A. Johnson+2

Independent Researcher, California, USA · Johns Hopkins University, Whiting School of Engineering, Baltimore, MD, USA

Temporal Reasoning in Language Models Language Modeling

Open paper AskPDF arXiv

Temporal decisions in language-model systems often depend on both symbolic task state and elapsed wall-clock time, such as cache expiration, job completion, quota resets, deadlines, or stale sessions. We study whether elapsed time can be supplied as a non-token, system-side scalar and composed with visible symbolic state by a frozen-backbone language model. We introduce ChronoState, a compositional temporal-state benchmark in which symbolic state appears in the prompt, elapsed seconds tau are supplied through a hidden chronometric-injection channel, and the model selects a forced-choice temporal action. Here, "hidden" means hidden from the user-visible token sequence, not from model computation. Using Qwen2.5-3B-Instruct as a frozen bf16 backbone with a 31-dimensional sinusoidal-plus-log time encoding, gated FiLM residual modulation, and a rank-8 LoRA action surface, hidden-time CI reaches 0.9305 +/- 0.0134 accuracy and 0.9410 +/- 0.0103 balanced accuracy. No-time and shuffled-time controls fall to 0.5511 +/- 0.0042 and 0.3323 +/- 0.0097, respectively, with high shuffled-time wrong-state consistency supporting causal dependence on the injected scalar within the trained distribution. Generalization remains strong for held-out templates, durations, and multi-constraint compositions, but held-out quota-family transfer is weak at 0.5065 +/- 0.0559, while a fair prompt+LoRA timestamp baseline reaches 0.9893 +/- 0.0052. Thus, ChronoState supports a narrow conclusion: hidden elapsed time can be composed with symbolic task state under direct supervision, but does not establish autonomous time tracking, broad unseen-family abstraction, or superiority over prompt-injected timestamps.

Abstract

Representation Geometry in Language Models

Explore similar work

ChronoState: Hidden Elapsed-Time Conditioning for Temporal-State Action Selection in Frozen-Backbone Language Models

Evaluating Temporal Consistency in Multi-Turn Language Models

What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?