Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16× less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.
Figures & tables
Figure 1: Semantic Tube improves data efficiency. (a) We hypothesize that error-free hidden state trajectories are geodesics, which are locally linear and approximated by the Semantic Tube. The dotted line depicts a trajectory distorted by training loss. Deviations perpendicular to the tube constitute noise , while the component along the geodesic represents the signal . (b) With our approach ( LNTP+LSTP ), accuracy shows a negligible drop when the training dataset is halved, and it matches full-dataset standard fine-tuning ( LNTP ) accuracy using only 161 of the training data. In contrast, LNTP degrades significantly when the dataset is halved.
Figure 2: Two hidden state trajectories with similar prefixes pass through the Voronoi cell of the “ researcher ” token at different locations, leading to different next hidden states and hence different next tokens. Since LNTP cannot guarantee that ht converges to ht∗ (optimal hidden state), ht can be misplaced on another geodesic. This leads to mode collapse (the red dotted line mistakenly continues the generation, misattributing Hinton’s Nobel Prize to an arbitrary person, or if the error deviates in the opposite direction and precludes a winner).
Figure 3: When the sentence aligns on a geodesic, the concept direction naturally aligns.
Figure 4: Loss landscape. (a) When LNTP plateaus, LSTP continues to decrease. Furthermore, minimizing LNTP does not automatically minimize LSTP . (b) Across a wide range of λ , increasing λ on a logarithmic scale reduces LSTP linearly, while LNTP remains unchanged.
Figure 5: Semantic Tube ( LNTP+LSTP , our approach) demonstrates superior performance across (a) datasets, (b) model families, and (c) model sizes compared to regular fine-tuning ( LNTP ) and LLM-JEPA ( LNTP+LJEPA ).
Suffix
Semantic Tube
Regular
LLM-JEPA
.*
88.5%
29.9%
68.9%
..
68.0%
28.0%
32.0%
Table 1: Accuracy on functionally equivalent regular expression suffixes “ .* ” and “ .. ”. Semantic Tube effectively captures nuanced preferences, whereas LLM-JEPA exhibits mode collapse by biasing towards “ .* ”, which is 35 × more prevalent in the training set than “ .. ”.
Figure 6: SVD decomposition demonstrating Semantic Tube’s polymorphism. (a) Without normalization, the SVD profile closely resembles regular fine-tuning. (b) With normalization, the SVD aligns with LLM-JEPA. Collectively, this indicates that Semantic Tube enforces a simple structure on the directions (normalized vectors) mapping Text to Code , while tolerating complexity in the unnormalized vectors. Note that the relative relationships among the base model, regular fine-tuning, and LLM-JEPA remain unchanged with or without normalization.
Figure 7: Impact of λ tuning on Llama-3 1B across various datasets. In most cases, peak performance is achieved within the range of 0.01 to 0.08 .
SYNTH
TURK
GSM8K
Spider
NQ
HS
0.02
0.04
0.005
0.04
0.16
0.02
Gemma2
Qwen3
R1 Dist
OLMo
OpenELM
0.005
0.02
0.04
0.01
0.04
Llama3 3B
Llama3 8B
0.01
0.0025
Table 2: Optimal λ values yielding maximum accuracy.
Figure 8: Ablation study. Semantic Tube (our approach) outperforms all variations. Within the Semantic Tube family, alternative configurations consistently degrade performance.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The same token vFrance directs the geodesic along different concept directions when appended to distinct prefixes, illustrating the necessity of the context-aware state difference ht−hs .
Figure 10: Geometric illustration for the proof of Lemma 3.2
Figure 11: The inference cone defines the probabilistic range of a Brownian motion, and its radius grows ∝σtt . A larger σt leads to a wider cone, which has a high probability of colliding with a token sequence trace that is far away (blue cone and green geodesic), while a smaller σt leads to a narrower cone that may only collide with a nearby trace (yellow cone and red geodesic). The dotted red and green fine lines are the Brownian motions confined by the yellow and blue cones, respectively.
Figure 12: Semantic Tube (our approach) and regular fine-tuning with 21 , 41 , 81 , 161 , and 321 dataset on (a) Llama3 3B and (b) Llama3 8B.
Regular Expressions
.([a-z]) ∣ ([AEIOUaeiou]) ∣ ([A-Za-z]).
.([A-Za-z]).([0-9])..
((dog)(.)).([AEIOUaeiou]).*
(dog).((truck) ∣ ([A-Z]) ∣ ([0-9])).
.(.)&([0-9])&(dog).
.(dog).((.))..*
Appendix
Table 3: Regular expression samples from the SYNTH dataset that end with either “ .* ” or “ .. ”, which are functionally equivalent.
Figure 13: Tuning λ for various configurations of (a) Semantic Tube, (b) Two Views, and (c) Mask. In all cases, the accuracy vs. λ curve is concave. We also observe that when λ exceeds the optimal value, accuracy declines rapidly while the standard deviation increases sharply, indicating that λ≪1 is preferred.
Two View
Mask
Curvature
LLM-JEPA2
1.14e-3
1.77e-3
3.04e-5
Two View
4.76e-3
5.10e-5
Mask
1.28e-4
Appendix
Table 4: Pairwise p -values comparing variation families. A cell is populated only if the mean accuracy of the row method exceeds that of the column method. p -values are computed using a paired, one-tailed t -test, restricted to the best-performing variant from each family.
Zero
Pred
Inst
2View
Pred
Mean
LLM-JEPA2
0.0534
2.03e-3
2.34e-4
2View+Warmup
0.265
0.0426
9.16e-6
+Zero
0.0185
5.56e-4
2View
0.0689
1.19e-6
+Pred
2.97e-4
+Pred
2.78e-4
Inst,Recov
Inst
Mask
Signed
Mask + all
0.0629
0.0159
4.02e-3
-Pred
2.34e-3
1.18e-3
Curvature
0.0368
Appendix
Table 5: Pairwise p -values comparing options within each variation family. A cell is populated only if the mean accuracy of the row option exceeds that of the column option. p -values are computed using a paired, one-tailed t -test. Values exceeding 0.05 are struck through.
Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168x more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4x for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3-12x over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2x more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.
Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA.
Jingnan Pu, Zi-En Fan, Feng Lian
School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China
Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.