The Order Is The Message

Jordan LeDoux

cs.LGMar 24, 2026

The Order Is The Message

Authors: Jordan LeDoux

Organizations: Independent Researcher

Abstract

In a controlled experiment on modular arithmetic ( $p = 9973$ ), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.

Figures & tables

Figure 1: The Stride and Fixed-Random strategies generalize quickly; the Random strategy memorizes but is unable to generalize within the compute budget; the Target strategy fails to either generalize or memorize, never improving beyond chance-level performance.

Figure 2: Both the Stride and Fixed-Random strategies accumulate frequency power early in training, and both end training with a similar amount of power concentrated in significant frequencies. The Stride strategy organizes much earlier than the Fixed-Random strategy, but the Fixed-Random achieve a higher peak concentration of frequency power.

Figure 3: The Embedding Spectral Entropy (blue), Decoder Spectral Entropy (orange), and Neuron Spectral Entropy (green) for all four strategies. This measures the amount of uniformity in weight distributions, with 1.0 being perfectly uniform, and 0.0 being maximally concentrated.

Figure 4: Validation accuracy plotted against embedding, decoder, and neuron spectral entropy for the two generalizing strategies. Both follow nearly identical curves despite building different frequency bases: generalization is a function of spectral concentration itself, independent of which frequencies the model concentrates into.

Figure 5: L2 norm of the ordering (blue) and content (orange) gradient components across training for Fixed-Random , Random , and Stride . Under the fixed-ordering strategies, the ordering norm peaks early and then declines as the model absorbs the coherent signal. Under Random , the ordering norm remains persistently large but incoherent, at approximately $2.8\times$ the content norm throughout training.

Figure 6: Cosine similarity between ordering and content gradient components across training. Both generalizing strategies show alignment dropping from $\sim$ 0.56 to $\sim$ 0.35 during early training, then hovering around 0.40 as both components converge toward the generalizing solution. Under Random , alignment begins similarly but collapses in late training as the model begins its slow approach toward grokking.

	Ordering	Ordering-Content	Epoch-to-Epoch	Final Test
	Mean	Mean
Strategy	Fraction	Alignment	Coherence	Accuracy
Random	87%	0.35	Incoherent	0.30%
Fixed-Random	84%	0.39	Coherent	99.5%
Stride	83%	0.41	Coherent	99.5%
Target	89%	0.25	Anti-coherent	0.01%

Table 1: Summary of ordering channel characteristics across the four strategies. The ordering fraction is similar in all cases; the strategies differ in temporal coherence and outcome. Details on the characterization of ”Coherent”, ”Incoherent”, and ”Anti-coherent” can be found in Appendix B.5

	Epochs to	Epochs to	Epochs to
	Finish	Finish	Finish
Strategy	WD=0.1	WD=0.05	WD=0.01
Stride	230	390	1375
Fixed-Random	230	381	1206
Random	253	437	1776

Table 2: Weight-decay ablation at

p=97

, averaged over 5 seeds per strategy. AdamW optimizer; training set size 2,500 (26.5% of problem space); embedding dimension 128; 1 layer; batch size 32. Fixed orderings degrade more gracefully than IID shuffling as regularization weakens.

Strategy	Path Efficiency
Stride	$4.94\times e^{-3}$
Fixed-Random	$3.71\times e^{-3}$
Random	$8.90\times e^{-4}$
Target	$7.54\times e^{-4}$

Table 3: The path efficiency of each strategy, measured as Net Displacement

/

Distance Traveled

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

	Mean	Mean	Mean	Ent-Content
	Entanglement	Content	Observed	Cosine
Strategy	Norm	Norm	Grad Norm	Similarity
Fixed-Random	122.5838	121.1522	3.4471	0.9991
Random	330.6589	328.0712	6.1070	0.9997
Stride	117.0194	115.6823	3.7507	0.9991
Target	60.3895	61.3590	11.2056	0.9773

Appendix

Table 4: Per-step Hessian decomposition of the observed gradient into entanglement (

\eta H_{B}\cdot g_{A}

) and content (

g_{B}(\theta)

) terms, averaged over epochs 0–500. The entanglement and content terms are nearly identical in magnitude and direction (cosine similarity

>

0.98), making the observed gradient a small residual: 30–55

\times

smaller in norm than either component for the non-adversarial strategies. The ordering signal resides in this residual. Values are means across all per-step burst measurements within the epoch range.

Metric	Formula	Description
loss	Cross-entropy loss	Training loss
train_acc	$100\times\text{correct}/\text{total}$	Training accuracy (%)
val_acc	$100\times\text{correct}/\text{total}$	Validation accuracy (%)
lr	From scheduler	Current learning rate
perplexity	$e^{\text{loss}}$	Exponentiated loss

Appendix

Table 5: Metrics from the training_metrics hook.

Metric	Formula	Description
total_norm	$\lVert\mathbf{g}\rVert_{2}$	L2 norm of full flattened gradient
max_component	$\max_{i}\lvert g_{i}\rvert$	Largest absolute gradient element
mean_component	$\operatorname{mean}_{i}(\lvert g_{i}\rvert)$	Mean absolute gradient element
norm_{layer}	$\lVert\mathbf{g}_{\ell}\rVert_{2}$	Per-layer gradient L2 norm

Appendix

Table 6: Metrics from the norms hook.

Metric	Formula	Description
cos_sim	$\cos(\mathbf{g}_{t},\,\mathbf{g}_{t-1})$	Cosine similarity between
		consecutive epoch gradients
angle_degrees	$\arccos(\text{cos\_sim})\cdot 180/\pi$	Angle between consecutive
		gradients

Appendix

Table 7: Metrics from the consecutive hook.

Metric	Formula	Description
gradient_variance	$\operatorname{mean}\bigl((\mathbf{g}_{i}-\bar{\mathbf{g}})^{2}\bigr)$	Element-wise variance
	across window	of gradients in window
mean_pairwise_cos	$\operatorname{mean}_{i<j}\cos(\mathbf{g}_{i},\mathbf{g}_{j})$	Mean pairwise cosine
		similarity in window
signal_to_noise	$\lVert\bar{\mathbf{g}}\rVert/\lVert\operatorname{std}(\mathbf{g})\rVert$	Signal-to-noise ratio
		of gradient window

Appendix

Table 8: Metrics from the variance hook.

Metric	Formula	Description
sv_concentration	$\operatorname{mean}\!\bigl(\sigma_{1}/\sum\sigma_{i}\bigr)$	Mean top-1 SV fraction
	across Q/K/V	of attention projections
effective_rank	$\operatorname{mean}\!\bigl((\sum\sigma)^{2}/\sum\sigma^{2}\bigr)$	Mean effective rank
		of Q/K/V weight matrices
top5_explained_var	$\operatorname{mean}\!\bigl(\sum_{i=1}^{5}\sigma_{i}^{2}/\sum\sigma^{2}\bigr)$	Mean variance explained
		by top-5 singular values

Appendix

Table 9: Metrics from the attention hook.

Metric	Formula	Description
low_freq_power	$\sum_{k<c}P_{k}/\sum P_{k}$	Fraction of power in
	$c=\max(p/20,10)$	lowest frequencies
spectral_entropy	$H(P)/\log p$	Normalized entropy of
		embedding power spectrum
peak_frequency	$\arg\max_{k\in[1,\,p/2)}P_{k}$	Dominant frequency index
		(excluding DC)

Appendix

Table 10: Metrics from the fourier hook.

Metric	Formula	Description
grad_velocity	$\lVert\mathbf{g}_{t}\rVert-\lVert\mathbf{g}_{t-1}\rVert$	First derivative of
		gradient norm
grad_acceleration	$v_{t}-v_{t-1}$	Second derivative of
		gradient norm
embedding_change	$\lVert\mathbf{E}_{t}-\mathbf{E}_{t-1}\rVert_{2}$	L2 distance between
		consecutive embeddings

Appendix

Table 11: Metrics from the phases hook.

Metric	Formula	Description
weight_norm/{layer}	$\lVert W_{\ell}\rVert_{2}$	Per-layer weight L2 norm
top_sv/{layer}	$\sigma_{1}(W_{\ell})$	Per-layer spectral norm
effective_rank/{layer}	$(\sum\sigma)^{2}/\sum\sigma^{2}$	Per-layer effective rank
grad_weight_align/{layer}	$\cos(\mathbf{g}_{\ell},W_{\ell})$	Per-layer gradient-weight
		cosine similarity
total_weight_norm	$\sqrt{\sum\lVert W_{i}\rVert^{2}}$	Whole-model weight L2 norm

Appendix

Table 12: Metrics from the weight_tracking hook.

Metric	Formula	Description
gradient_sparsity	$\lvert\{i:\lVert\mathbf{g}_{i}\rVert<0.1\bar{n}\}\rvert/p$	Fraction of rows with
		small gradients
gradient_gini	Gini index of row norms	Inequality of gradient
		distribution across tokens
stride_group_variance	$\operatorname{Var}(\bar{n}_{g}\text{ per stride group})$	Variance of mean norm
		across stride groups

Appendix

Table 13: Metrics from the token_gradient hook.

Metric	Formula	Description
grad_cossim_to_solution/{layer}	$\cos(-\nabla_{\ell}L,\;\theta_{\text{ref}}-\theta_{\text{prev}})$	Per-layer gradient
		alignment to solution
disp_cossim_to_solution/{layer}	$\cos(\Delta\theta_{\ell},\;\theta_{\text{ref}}-\theta_{\text{prev}})$	Per-layer displacement
		alignment to solution
overall_grad_cossim_to_solution	$\cos(\text{cat}(-\nabla L),\;\text{cat}(\Delta_{\text{ref}}))$	All-parameter gradient
		alignment to solution

Appendix

Table 14: Metrics from the gradient_projection hook.

Metric	Formula	Description
dims_for_90pct	$\min k$ s.t. $\sum_{i=1}^{k}\sigma_{i}^{2}/\sum\sigma^{2}\geq 0.9$	SVD dimensions for 90%
		explained variance
participation_ratio	$(\sum\sigma^{2})^{2}/\sum\sigma^{4}$	Effective dimensionality
		of gradient subspace
top_sv_ratio	$\sigma_{1}/\sum\sigma_{i}$	Dominance of leading
		singular value

Appendix

Table 15: Metrics from the subspace_gradient_info hook.

Metric	Formula	Description
relative_delta	$\lVert\Delta\theta\rVert/\lVert\theta_{\text{old}}\rVert$	Fractional parameter change
absolute_delta	$\lVert\theta_{\text{new}}-\theta_{\text{old}}\rVert_{2}$	Absolute parameter change
param_norm	$\lVert\theta_{\text{new}}\rVert_{2}$	Current parameter L2 norm

Appendix

Table 16: Metrics from the parameter_delta hook.

Metric	Formula	Description
path_length	$\sum_{t}\lVert\theta_{t+1}-\theta_{t}\rVert_{2}$	Cumulative distance traveled
		in parameter space
net_displacement	$\lVert\theta_{t}-\theta_{0}\rVert_{2}$	Euclidean distance from
		initial parameters
path_efficiency	$\lVert\theta_{t}-\theta_{0}\rVert/\text{path\_length}$	Displacement-to-path ratio;
		$1=$ straight line

Appendix

Table 17: Metrics from the path_length hook.

Metric	Formula	Description
lag_{n}	$\operatorname{mean}\!\bigl(\cos(\mathbf{g}_{t},\mathbf{g}_{t-n})\bigr)$	Mean cosine similarity
	$n\in\{1,2,5,10,20,50\}$	at lag $n$ steps
autocorrelation_mean	$\operatorname{mean}(\text{lag}_{k}\text{ for all }k)$	Overall temporal coherence
		of gradients
efficiency_{w}	$\lVert\sum_{i}^{w}\mathbf{g}_{i}\rVert/\sum\lVert\mathbf{g}_{i}\rVert$	Accumulation efficiency
	$w\in\{2,5,10,20,50\}$	over $w$ -step window

Appendix

Table 18: Metrics from the batch_dynamics hook.

Metric	Formula	Description
loss_mean	$\bar{L}$ over emission period	Mean loss
loss_std	$\operatorname{std}(L_{t})$	Std of per-step loss
loss_volatility	$\operatorname{std}(L)/\lvert\bar{L}\rvert$	Coefficient of variation
		of loss
loss_autocorrelation	$\operatorname{corr}(L_{t},L_{t+1})$	Lag-1 autocorrelation
		of loss sequence

Appendix

Table 19: Metrics from the training_diagnostics hook.

Metric	Formula	Description
entanglement_norm	$\lVert\eta\,H_{B}\,\mathbf{g}_{A}\rVert_{2}$	L2 norm of the
		entanglement term
content_norm	$\lVert\mathbf{g}_{B}+\eta\,H_{B}\,\mathbf{g}_{A}\rVert_{2}$	L2 norm of the
		ordering-invariant term
observed_grad_norm	$\lVert\mathbf{g}_{B}\rVert_{2}$	Observed batch gradient norm
entanglement_energy_ratio	$\lVert\text{ent}\rVert^{2}/\lVert\mathbf{g}_{B}\rVert^{2}$	How much ordering

Appendix

Table 20: Metrics from the hessian hook.

\eta H_{B}\mathbf{g}_{A}

is computed via finite-difference Hv product.

Metric	Formula	Description
counterfactual_mean_norm	$\lVert\bar{\mathbf{g}}_{\text{cf}}\rVert_{2}$	Norm of mean
		shuffled-epoch gradient
content_component_norm	$\lVert\operatorname{proj}(\mathbf{g}\to\bar{\mathbf{g}}_{\text{cf}})\rVert_{2}$	Norm of ordering-invariant
		content component
ordering_component_norm	$\lVert\mathbf{g}-\text{content}\rVert_{2}$	Norm of ordering-specific
		component

Appendix

Table 21: Metrics from the counterfactual hook. Decomposition uses

K

shuffled training epochs.

\Delta_{\text{ref}}=\theta_{\text{ref}}-\theta_{\text{prev}}

Tier 1: Direction-agnostic
Metric	Formula	Description
momentum_grad_cossim	$\cos(\mathbf{m},\,\mathbf{g})$	First moment vs. current
		gradient
amplification_ratio	$\lVert\text{update}\rVert/\lVert\eta\,\mathbf{g}\rVert$	Adam update norm vs.
		raw SGD update norm
update_deflection	$\lVert\text{update}_{\perp}\rVert/\lVert\text{update}\rVert$	Fraction of update

Appendix

Table 22: Metrics from the adam_dynamics hook. Adam update is

\text{lr}\cdot\hat{m}/(\sqrt{\hat{v}}+\epsilon)

; weight decay is excluded to isolate adaptive dynamics.

Stride $s$	Predicted $F=\lfloor p/s\rceil$	Observed $F$
50	199	199
99	101	101
150	66	66

Appendix

Table 23: Predicted vs. observed fundamental frequency

Figure 7: Measured amplification ratio of AdamW updates per strategy.

Figure 8: Measured update deflection of AdamW updates per strategy. All strategies experience high deflection.

Figure 9: Measured effective learning-rate coefficient of variation.

Figure 10: The autocorrelation between batch gradients of the Target strategy are so large that they make the details of the other strategies difficult to examine on this scale.

Figure 11: Removing the Target strategy allows us to see the behavior of Stride , which peaks during the critical learning period.

Figure 12: Removing both Target and Stride allows us to see just how little ordering information and consistency was necessary for generalization to develop in the Fixed-Random strategy compared to the Random strategy.

Data-Efficient Learning

Data-efficient learning develops methods that achieve accurate, robust models with fewer labeled examples, training samples, or interactions.

See all 241 papers

Sep 23, 2026

Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score

"Train a model on the same problems written under two incompatible conventions, both correct, and ask what the ordering of that data writes into the parameters. The learning-rate schedule is not a background condition for that question. It is the averaging operator, and it decides the answer. We prove a bound in which the arrangement and the schedule enter the ordering effect as separate multiplied factors: the arrangement only as a block period, the schedule only as how much weight the endpoint can place on any one moment of the run. A decaying schedule cannot put a large step size and an uncontracted remainder at the same moment; a constant one does exactly that at the last step. That decay moderates ordering effects has been reported in pretraining; the mechanism, the separation, and a controlled measurement of both halves are ours. Ten orderings of one corpus, one budget, everything but the path held fixed, run twice under families differing in lr_scheduler_type and nothing else: at a constant rate the interior spans 0.2221 in allocation, 11.63 contrast floors, monotone in how blocked the arrangement is. Under the single cosine every published arm uses, the same ten arms occupy two distinguishable states where their own resolution would allow about ten. "Order matters" and "order does not matter" are the two ends of one knob. What the path writes is which convention the model commits to, and no exact-match benchmark can see it. Across twelve arms acc_A+acc_B is constant to within 9.7% while the allocation share runs 0.04 to 0.87, so the 12.29-sigma arrangement switch this paper measures is exactly zero under a convention-agnostic metric. That conservation is quoted from the decayed family throughout, the constant-rate one being a noisier place to read it. Marking the convention in the prompt collapses the switch and reaches 87.5% of the union ceiling."

Wenhui Chen

University of Macau

Learning Rate Scheduling

Ask PDF

Jun 28, 2026

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a single seed to flip a close A/B comparison. We isolate this mechanism and derive a fit-free way to size the noise it produces. For a memoryless optimizer, reordering an equal multiset has no first-order endpoint term; the leading local contrast is the

O(η^2)

gradient bracket. Fixed-clock optimizers such as AdamW are different. Their moment buffers, preconditioner state, and de-biasing counters advance with the step index rather than with the learning-rate-scaled time

τ=ηk

, so the same gradient can receive a position-dependent endpoint weight. For any fixed finite measurement window, a lifted-state expansion gives an

O(η)

equal-multiset contrast whenever the first-order replay coefficient is nonzero, while regular and clock-matched controls remain

O(η^2)

; a bare fixed-

β

momentum buffer is already enough. A bitwise-deterministic replay from one warmed optimizer state isolates the mechanism, giving order-variance slopes 1.83 for AdamW, 2.00 for fixed-

β

momentum, and 4.00 for SGD; matching the memory clock to

τ

restores the regular exponent. For AdamW with a frozen preconditioner, the same impulse-weight kernel gives a closed-form asymptotic order-variance floor after the local potentials are measured, with no fitted coefficients. The result is local to the measurement window (independent reshuffling can average the channel across windows), but it yields order-noise error bars, positional attribution weights, and a seed-budget criterion for fine-tuning comparisons.

John Sweeney

Stochastic Optimization Fine-Tuning Adaptive Gradient Methods

Ask PDF

May 6, 2026

Order-based Rehearsal Learning

When a machine learning (ML) model forecasts an undesired event, one often seeks a decision to avoid it, known as the avoiding undesired future (AUF) problem. Many rehearsal learning methods have been proposed for AUF, but they rely on an underlying graph structure; learning such a graph from observational data is challenging and can incur substantial estimation error. In this work, we demonstrate that the order structure can be sufficient for AUF decision-making, and propose the first order-based rehearsal learning method. Although an order is less informative than a graph, it can be sufficient to identify the influence of decisions from observational data, suggesting that learning the entire graph is not always necessary. To learn the order, we develop an information-theoretic method that imposes no restrictions on the form of structural functions or the type of noise distributions. For AUF decision-making, we construct an order-based sampler to approximate the influence of decisions and, combined with a surrogate objective for maximizing the post-decision success probability, reduce the AUF task to a differentiable optimization problem. Experiments show that our order learning method outperforms existing methods, and that our AUF approach not only surpasses methods relying on learned graphs or learned orders, but also matches or even exceeds oracle baselines that are given the true graph.

Yu-Xuan Tao, Tian-Zuo Wang, Zhi-Hua Zhou

National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artifcial Intelligence, Nanjing University, China

Causal Discovery Causal Structure Learning

Ask PDF

Abstract

Data-Efficient Learning

Explore similar work

Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

Order-based Rehearsal Learning