Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16× less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.
Figures & tables
Figure 1: Semantic Tube improves data efficiency. (a) We hypothesize that error-free hidden state trajectories are geodesics, which are locally linear and approximated by the Semantic Tube. The dotted line depicts a trajectory distorted by training loss. Deviations perpendicular to the tube constitute noise , while the component along the geodesic represents the signal . (b) With our approach ( LNTP+LSTP ), accuracy shows a negligible drop when the training dataset is halved, and it matches full-dataset standard fine-tuning ( LNTP ) accuracy using only 161 of the training data. In contrast, LNTP degrades significantly when the dataset is halved.
Figure 2: Two hidden state trajectories with similar prefixes pass through the Voronoi cell of the “ researcher ” token at different locations, leading to different next hidden states and hence different next tokens. Since LNTP cannot guarantee that ht converges to ht∗ (optimal hidden state), ht can be misplaced on another geodesic. This leads to mode collapse (the red dotted line mistakenly continues the generation, misattributing Hinton’s Nobel Prize to an arbitrary person, or if the error deviates in the opposite direction and precludes a winner).
Figure 3: When the sentence aligns on a geodesic, the concept direction naturally aligns.
Figure 4: Loss landscape. (a) When LNTP plateaus, LSTP continues to decrease. Furthermore, minimizing LNTP does not automatically minimize LSTP . (b) Across a wide range of λ , increasing λ on a logarithmic scale reduces LSTP linearly, while LNTP remains unchanged.
Figure 5: Semantic Tube ( LNTP+LSTP , our approach) demonstrates superior performance across (a) datasets, (b) model families, and (c) model sizes compared to regular fine-tuning ( LNTP ) and LLM-JEPA ( LNTP+LJEPA ).
Suffix
Semantic Tube
Regular
LLM-JEPA
.*
88.5%
29.9%
68.9%
..
68.0%
28.0%
32.0%
Table 1: Accuracy on functionally equivalent regular expression suffixes “ .* ” and “ .. ”. Semantic Tube effectively captures nuanced preferences, whereas LLM-JEPA exhibits mode collapse by biasing towards “ .* ”, which is 35 × more prevalent in the training set than “ .. ”.
Figure 6: SVD decomposition demonstrating Semantic Tube’s polymorphism. (a) Without normalization, the SVD profile closely resembles regular fine-tuning. (b) With normalization, the SVD aligns with LLM-JEPA. Collectively, this indicates that Semantic Tube enforces a simple structure on the directions (normalized vectors) mapping Text to Code , while tolerating complexity in the unnormalized vectors. Note that the relative relationships among the base model, regular fine-tuning, and LLM-JEPA remain unchanged with or without normalization.
Figure 7: Impact of λ tuning on Llama-3 1B across various datasets. In most cases, peak performance is achieved within the range of 0.01 to 0.08 .
SYNTH
TURK
GSM8K
Spider
NQ
HS
0.02
0.04
0.005
0.04
0.16
0.02
Gemma2
Qwen3
R1 Dist
OLMo
OpenELM
0.005
0.02
0.04
0.01
0.04
Llama3 3B
Llama3 8B
0.01
0.0025
Table 2: Optimal λ values yielding maximum accuracy.
Figure 8: Ablation study. Semantic Tube (our approach) outperforms all variations. Within the Semantic Tube family, alternative configurations consistently degrade performance.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The same token vFrance directs the geodesic along different concept directions when appended to distinct prefixes, illustrating the necessity of the context-aware state difference ht−hs .
Figure 10: Geometric illustration for the proof of Lemma 3.2
Figure 11: The inference cone defines the probabilistic range of a Brownian motion, and its radius grows ∝σtt . A larger σt leads to a wider cone, which has a high probability of colliding with a token sequence trace that is far away (blue cone and green geodesic), while a smaller σt leads to a narrower cone that may only collide with a nearby trace (yellow cone and red geodesic). The dotted red and green fine lines are the Brownian motions confined by the yellow and blue cones, respectively.
Figure 12: Semantic Tube (our approach) and regular fine-tuning with 21 , 41 , 81 , 161 , and 321 dataset on (a) Llama3 3B and (b) Llama3 8B.
Regular Expressions
.([a-z]) ∣ ([AEIOUaeiou]) ∣ ([A-Za-z]).
.([A-Za-z]).([0-9])..
((dog)(.)).([AEIOUaeiou]).*
(dog).((truck) ∣ ([A-Z]) ∣ ([0-9])).
.(.)&([0-9])&(dog).
.(dog).((.))..*
Appendix
Table 3: Regular expression samples from the SYNTH dataset that end with either “ .* ” or “ .. ”, which are functionally equivalent.
Figure 13: Tuning λ for various configurations of (a) Semantic Tube, (b) Two Views, and (c) Mask. In all cases, the accuracy vs. λ curve is concave. We also observe that when λ exceeds the optimal value, accuracy declines rapidly while the standard deviation increases sharply, indicating that λ≪1 is preferred.
Two View
Mask
Curvature
LLM-JEPA2
1.14e-3
1.77e-3
3.04e-5
Two View
4.76e-3
5.10e-5
Mask
1.28e-4
Appendix
Table 4: Pairwise p -values comparing variation families. A cell is populated only if the mean accuracy of the row method exceeds that of the column method. p -values are computed using a paired, one-tailed t -test, restricted to the best-performing variant from each family.
Zero
Pred
Inst
2View
Pred
Mean
LLM-JEPA2
0.0534
2.03e-3
2.34e-4
2View+Warmup
0.265
0.0426
9.16e-6
+Zero
0.0185
5.56e-4
2View
0.0689
1.19e-6
+Pred
2.97e-4
+Pred
2.78e-4
Inst,Recov
Inst
Mask
Signed
Mask + all
0.0629
0.0159
4.02e-3
-Pred
2.34e-3
1.18e-3
Curvature
0.0368
Appendix
Table 5: Pairwise p -values comparing options within each variation family. A cell is populated only if the mean accuracy of the row option exceeds that of the column option. p -values are computed using a paired, one-tailed t -test. Values exceeding 0.05 are struck through.