cs.LGOct 13, 2026

Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation

Authors: Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa, Yutaka Matsuo

Organizations: The University of Tokyo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle.

Figures & tables

Figure 1: NDA at a glance. (Left) Scale-control mechanisms in deep learning, including normalization, residual scaling, attention temperature, loss weighting, and adaptive updates, manage scale at different interfaces of a Transformer. Unit Violation Indices (UVI) flag deviations from the intended scale regime, locating the interface where a scale mismatch arises. (Middle) NDA expresses these mechanisms through unit-consistent operations and a factorization into scales and canonical coordinates. (Right) The Gauge-Canonical Transformer (GCT) builds the whole network on this factorization. Every block computes on dimensionless coordinates (blue), ensuring matching units at every interface, including attention and multilayer perceptron (MLP) branches. The scale (gold) is reattached only when an output needs units.
Figure 2: Final validation loss as a function of peak learning rate for primary-seed 51M models trained on 600M tokens. GCT and nGPT retain low loss at high rates that worsen the other three architectures, showing a wider useful learning-rate range.
architectureL=24L=24L=48L=48
Pre-LN4.3974.598
LLaMA-style4.3224.513
QK-Norm4.2064.322
nGPT3.9964.243
GCT (ours)3.9733.903
GCT without P⊥P^{\perp}3.9833.938
Table 1: Validation loss in fixed-budget depth transfer with the primary seed, 400M tokens, and hyperparameters transferred unchanged from L=8L=8 . GCT achieves the lowest loss at both depths, and adding the GCT projection also improves nGPT at L=48L=48 .
  weights and activationsstream h→int8h\to\mathrm{int8}stream h→int4h\to\mathrm{int4}
architecturebaseW4W8A8tensortokentensortoken
Pre-LN2.759+0.045+0.028+3.018+0.049+6.390+7.101
LLaMA-style2.744+0.032+0.348+5.513+0.030+7.117+5.474
QK-Norm2.737+0.029+0.833+5.426+0.021+8.074+5.230
nGPT2.751+0.038+0.075+0.032+0.008+19.083+4.661
GCT (ours)2.756+0.028+0.079+0.009+0.001+9.517+0.739
Table 2: PTQ at 1B with d=1792d=1792 , L=24L=24 , 4B tokens, and the primary seed. Base is the unquantized 16-bit brain floating-point (bf16) loss on a fixed 32-batch subset, while other entries are loss increases in nats. Per-channel 4-bit weight quantization (W4) changes only weights. In 8-bit weight-and-activation quantization (W8A8), linear-input activations use static scales. Stream columns use static per-layer tensor scales or dynamic per-token scales at every block boundary. The logit head remains in full precision. GCT has the smallest stream loss increase with per-tensor int8 and per-token int4 quantization, while per-tensor int4 causes large loss increases for every architecture.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

SymbolMeaning
Units and the rescaling group
GGdeclared rescaling group (R+)K(\mathbb{R}_{+})^{K} of the graph
gg , ccgroup element; a single positive rescaling factor
ρv\rho_{v}neural unit of graph node vv , a representation of GG
χu\chi_{u}scalar character g↦∏kgkukg\mapsto\prod_{k}g_{k}^{u_{k}} with exponent uu
u(x)u(x)unit type of xx ; u(x)=1u(x)=1 means dimensionless
Appendix
Table 3: List of notation. Recurring symbols are grouped by the part of the calculus that introduces each symbol. A bar denotes a dimensionless canonical coordinate, and a star denotes a target value of a diagnostic.
architecturevalidation loss
Pre-LN3.172±0.0023.172\pm 0.002
LLaMA-style3.175±0.0033.175\pm 0.003
QK-Norm3.147±0.002\mathbf{3.147\pm 0.002}
NormFormer3.198±0.0083.198\pm 0.008
LayerScale3.167±0.0033.167\pm 0.003
nGPT3.170±0.0033.170\pm 0.003
Appendix
Table 4: Clean validation loss for the matched 124M models trained on 2.75B tokens. Each row reports the mean and standard deviation over three seeds under the shared recipe. Values are rounded to three decimal places. A displayed standard deviation of 0.0000.000 denotes variation below that precision.
 peak learning rate
architecture3×10−43\times 10^{-4}6×10−46\times 10^{-4}1.2×10−31.2\times 10^{-3}2.4×10−32.4\times 10^{-3}4.8×10−34.8\times 10^{-3}9.6×10−39.6\times 10^{-3}
Pre-LN4.2323.9433.7503.8763.881abort
LLaMA-style4.2483.8923.7143.8023.883abort
QK-Norm4.1533.8403.7003.7403.8793.897
nGPT4.3833.8933.6493.5633.5073.500
GCT (ours)4.3733.8853.6583.5483.5173.513
Appendix
Table 5: Final validation loss across a 16×16\times learning-rate range for 51M models trained on 600M tokens. The additional octave at 9.6×10−39.6\times 10^{-3} is reported wherever training runs to completion. abort marks a run terminated early because the run’s UVI score exceeded the preset threshold, as specified in App. H .
Seed variability at 51M
architectureval loss
Pre-LN3.9405 ±\pm 0.0039
LLaMA-style3.9024 ±\pm 0.0167
QK-Norm3.8445 ±\pm 0.0037
nGPT3.8934 ±\pm 0.0004
GCT (ours)3.8944 ±\pm 0.0096
Appendix
Table 6: Seed variability and GCT component ablations. Both tables follow the Table 5 protocol at learning rate 6×10−46\times 10^{-4} . The seed table reports mean ±\pm standard deviation over three seeds, while the ablations use the primary seed.
Figure 3: Measured residual steps and alignments (αℓ,cℓ)(\alpha_{\ell},c_{\ell}) during depth transfer to L=48L=48 , six times the tuning depth. Results use primary-seed fixed-budget probes. The shared legend below the panels names every architecture. (a) Measured step size against alignment. Grey contours give the per-sub-block stream-scale gain sℓ+1/sℓ=(1+αℓ2+2αℓcℓ)1/2s_{\ell+1}/s_{\ell}=(1+\alpha_{\ell}^{2}+2\alpha_{\ell}c_{\ell})^{1/2} of Eq. 6 , the dashed ×1\times 1 contour being the scale-preserving locus, and the dotted line marks α⋆=1/2L\alpha^{\star}=1/\sqrt{2L} . (b) Removing P⊥P^{\perp} from GCT increases alignment, whereas adding the projection to nGPT reduces alignment and changes loss from 4.2434.243 to 3.9683.968 . (c) Tangential–radial step decomposition, with the final validation loss of each run in the right-hand column. Points are medians over the final three measurements. Hyperparameters transfer unchanged from L=8L=8 .
Figure 4: UVI fingerprint for matched 51M stress runs at seed 1337 and 600M tokens, without warmup or clipping, at peak learning rate 9.6×10−39.6\times 10^{-3} . (a) QK-Norm repairs attention variance and entropy, while (b) QK-Norm’s residual RMS still reaches values between 1.6×1021.6\times 10^{2} and 3.3×1023.3\times 10^{2} . GCT keeps both readouts controlled. The annotations report final-step statistics: the layer median of vv , the mean attention entropy as a fraction of the maximum attention entropy, and the across-layer range of the stream RMS. This figure is a single-seed localization case, while Figure 2 and Table 5 provide the sweep evidence.
  124M val loss 
architecture51M-selected rateat 6×10−46\times 10^{-4}at selected rateΔ\Delta
Pre-LN1.2×10−31.2\times 10^{-3}3.1723.119−0.052-0.052
LLaMA-style3.1783.105−0.072-0.072
QK-Norm3.1493.091−0.058-0.058
nGPT4.8×10−34.8\times 10^{-3}3.1713.043−0.127-0.127
GCT (ours)3.1703.045 ±\pm 0.002−0.125-0.125
Appendix
Table 7: Sweep-to-scale transfer. Each architecture selects a learning rate from the original five-point 16×16\times grid in Table 5 and applies the selected learning rate unchanged at 124M. The 9.6×10−39.6\times 10^{-3} stress point is excluded. The selected-rate GCT entry reports mean ±\pm standard deviation over two seeds, while other loss entries use the primary seed. Δ\Delta is computed from unrounded values relative to the primary-seed result at 6×10−46\times 10^{-4} .
Figure 5: Validation loss under internal rescaling h→chh\to ch at the embedding output and mid-depth, with fp32 block error. The intervention changes the hidden state’s magnitude while preserving the state’s direction. GCT and nGPT preserve validation loss, with block invariance errors near the numerical floor, showing insensitivity to the imposed magnitude change. UCT loses invariance at c=10−2c=10^{-2} because the hidden state shrinks while the small constants in UCT’s normalization denominators and the stored evaluation-time scale estimates remain fixed. The fixed constants account for a larger share of the normalization denominators, and the stored scales no longer match the activations. These mismatches prevent exact cancellation of the rescaling. Baselines with unnormalized residual streams also degrade.
  rescaled at embeddingrescaled at mid-depth 
architecturebasec=0.1c=0.1c=10c=10c=0.1c=0.1c=10c=10eq error
Pre-LN3.1187.9013.96410.5815.2393.7×10−13.7\times 10^{-1}
LLaMA-style3.1267.3174.8848.1185.5923.4×10−13.4\times 10^{-1}
QK-Norm3.0987.4744.2998.9095.3334.0×10−14.0\times 10^{-1}
NormFormer3.1393.1413.27710.1535.3704.7×10−14.7\times 10^{-1}
LayerScale3.1208.72322.86011.4387.8753.3×10−13.3\times 10^{-1}
Appendix
Table 8: Validation loss of the trained 124M models at the c=0.1c=0.1 and c=10c=10 points of Figure 5 , together with block equivariance error at c=10c=10 .
 attentionMLP
architectureqqkkvvoutinupout
Pre-LN4.1×10−34.1\times 10^{-3}4.1×10−34.1\times 10^{-3}3.5×10−33.5\times 10^{-3}3.5×10−33.5\times 10^{-3}2.6×10−22.6\times 10^{-2}–1.6×10−21.6\times 10^{-2}
LLaMA-style4.2×10−34.2\times 10^{-3}4.2×10−34.2\times 10^{-3}1.2×10−21.2\times 10^{-2}1.2×10−21.2\times 10^{-2}2.3×10−22.3\times 10^{-2}1.6×10−21.6\times 10^{-2}1.6×10−21.6\times 10^{-2}
QK-Norm6.4×10−9†6.4\times 10^{-9}\dagger1.5×10−8†1.5\times 10^{-8}\dagger4.6×10−34.6\times 10^{-3}4.6×10−34.6\times 10^{-3}2.1×10−22.1\times 10^{-2}1.3×10−21.3\times 10^{-2}1.3×10−21.3\times 10^{-2}
NormFormer4.3×10−34.3\times 10^{-3}4.3×10−34.3\times 10^{-3}2.4×10−7†2.4\times 10^{-7}\dagger2.4×10−7†2.4\times 10^{-7}\dagger9.2×10−39.2\times 10^{-3}–1.3×10−21.3\times 10^{-2}
LayerScale4.5×10−34.5\times 10^{-3}4.5×10−34.5\times 10^{-3}6.7×10−36.7\times 10^{-3}6.7×10−36.7\times 10^{-3}2.0×10−22.0\times 10^{-2}1.1×10−21.1\times 10^{-2}1.1×10−21.1\times 10^{-2}
Appendix
Table 9: The Prop. 6 fingerprint on the trained 124M models in fp32 evaluation mode. Entries report max⁡ℓ∣⟨Wℓ,∇WℓL⟩∣\max_{\ell}|\langle W_{\ell},\nabla_{W_{\ell}}\mathcal{L}\rangle| in nats per ee -fold for each weight-matrix role. The symbol †\dagger marks a cell predicted to be an exact null by the architecture’s typing, with the proof in App. G.10 and the rolewise predictions in App. H.6 . Every marked cell sits at the fp32 floor, while every other cell is orders of magnitude above that floor. Equal-pairing roles agree as the theory requires for qq and kk , for vv and the attention output, and for the up-projection and MLP output. A double dash denotes an absent role because a GELU MLP has no separate up-projection.
  weights and activationsstream h→int8h\to\mathrm{int8}stream h→int4h\to\mathrm{int4}
architecturebaseW4W8A8tensortokentensortoken
Pre-LN3.118+0.067+0.016+0.085+0.002+6.723+0.912
LLaMA-style3.126+0.063+0.270+1.058+0.001+6.212+1.044
QK-Norm3.098+0.061+0.237+1.099+0.001+7.715+0.386
NormFormer3.139+0.125+0.234+0.034+0.001+4.164+0.761
LayerScale3.120+0.061+0.178+0.896+0.002+6.214+0.691
Appendix
Table 10: PTQ of the trained 124M models. Values are changes in validation loss relative to bf16, measured in nats, and positive values are worse. The 8-bit weight-only setting is not tabulated because that setting changes loss by 0.0000.000 nats for every architecture, both here and at 1B. W8A8 uses static absmax calibration at every linear input. The h→int8h\to\mathrm{int8} and h→int4h\to\mathrm{int4} columns quantize the residual stream at every block boundary with per-layer static tensor scales or per-token scales. The logit head remains in full precision.
Figure 6: Quantization degradation by scheme on a logarithmic scale. Per-tensor int8 on the residual stream separates the architectures by orders of magnitude. Per-token scales remove token-to-token scale spread from the quantizer’s relative resolution and substantially reduce the int8 gap.
  int8int4
architecturebf16 (%)tensortokentoken
Pre-LN48.6−20.5-20.5−1.9-1.9−22.7-22.7
LLaMA-style48.7−22.7-22.7−0.8-0.8−22.6-22.6
QK-Norm49.4−19.6-19.6−0.5-0.5−23.1-23.1
nGPT49.0−0.8-0.8−0.3-0.3−17.4-17.4
GCT (ours)49.0−0.1-0.1+0.2+0.2−7.4-7.4
Appendix
Table 11: Downstream accuracy under residual-stream quantization on the 1B models trained independently with a second seed. We report zero-shot accuracy with the harness formatting for LAMBADA, HellaSwag, PIQA, and ARC-Easy. The stream is stored at the same block-boundary sites and granularities as in Table 2 , using dynamic scales without calibration, and the evaluation is inference-only, with no retraining or fine-tuning. The bf16 column gives mean accuracy over the four tasks in percent, and the remaining columns give accuracy changes in points, where negative values indicate lower accuracy.
  int8int4
modelκdepth\kappa_{\mathrm{depth}}tensortokentoken
GPT-2 (124M)4848+5.79+5.79+0.008+0.008+4.25+4.25
GPT-2 (355M)236236+7.91+7.91+0.046+0.046+8.02+8.02
GPT-2 (774M)223223+9.03+9.03+0.034+0.034+6.02+6.02
GPT-2 (1.5B)596596+7.30+7.30+0.035+0.035+8.05+8.05
Appendix
Table 12: External-validity audit on off-the-shelf GPT-2. The OpenAI GPT-2 family uses Pre-LN and was trained by a third party on WebText. We evaluate the family on the same validation bins and residual-stream fake-quantization hooks as our trained models, using dynamic scales, with loss increases reported in nats relative to each unquantized model. The quantity κdepth\kappa_{\mathrm{depth}} is the ratio between the maximum and minimum mean residual-stream RMS across block inputs, not the within-tensor spread of Prop. 5 . Every size has a large per-tensor loss increase and a near-lossless per-token result. For comparison, our 124M GCT model has κdepth≈1\kappa_{\mathrm{depth}}\approx 1 and a ++ 0.001-nat dynamic-int8 loss increase.
 model size
architecture124M350M1.0B
Pre-LN+0.085+0.085+1.009+1.009+3.018+3.018
LLaMA-style+1.058+1.058+2.453+2.453+5.513+5.513
QK-Norm+1.099+1.099+0.325+0.325+5.426+5.426
nGPT+0.005+0.005+0.010+0.010+0.032+0.032
GCT (ours)+0.001+0.001+0.003+0.003+0.009+0.009
Appendix
Table 13: Calibrated per-site, per-tensor int8 residual-stream quantization along the 124M, 350M, and 1B ladder. Columns report one trained model per architecture, the primary seed at 124M and 1B and the second seed at 350M. nGPT and GCT remain near the int8 floor. The largest baseline loss increase grows with parameter count, although individual trajectories need not be monotone.
Figure 7: Table 13 on log–log axes. Along the tested model-size ladder, the worst-baseline increase reaches +5.51+5.51{} nats at 1B, while nGPT and GCT remain near the int8 floor.
 GCTPre-LN
width6×10−46\times 10^{-4}4.8×10−34.8\times 10^{-3}μ\mu P6×10−46\times 10^{-4}1.2×10−31.2\times 10^{-3}4.8×10−34.8\times 10^{-3}μ\mu P
d=256d=2564.3663.7733.7684.4194.1664.3074.186
d=512d=5123.8853.5173.5173.9433.7503.8813.750
d=1024d=10243.5903.3453.3513.6133.6175.6523.549
Appendix
Table 14: Width transfer and learning-rate-only μ\mu P composition for the 51M-centered family, with L=8L=8 , dh=64d_{h}=64 , and the primary seed under the Table 5 token budget. The standard-parameterization (SP) columns use the listed learning rate. The μ\mu P columns scale hidden-matrix learning rates by 512/d512/d from the architecture’s d=512d=512 -tuned rate, namely 4.8×10−34.8\times 10^{-3} for GCT and 1.2×10−31.2\times 10^{-3} for Pre-LN. At the tuning width, the rule reduces to SP.
  robustness
architecturecleannoisecontrastbrightin-scale
Pre-LN69.8137.4963.2768.4551.58
QK-Norm70.6836.9264.8069.5856.00
nGPT73.9834.3468.9273.0063.54
Corrective UCT (ours)73.2833.6668.0072.2562.25
GCT (ours)74.1734.3669.2372.9364.13
Appendix
Table 15: CIFAR-100 from scratch with the DeiT-Ti recipe for 300 epochs. Entries report top-1 accuracy in percent. Robustness columns average over shift grids with noise σ∈{0.05,0.1}\sigma\in\{0.05,0.1\} , contrast {0.5,2.0}\{0.5,2.0\} , brightness {0.7,1.3}\{0.7,1.3\} , and input scale {0.25,0.5,2,4}\{0.25,0.5,2,4\} .
Figure 8: ViT robustness grid on CIFAR-100 accompanying Table 15 . The figure reports top-1 accuracy under the photometric and input-scale shift grid.
  robustness
architecturecleannoisecontrastin-scale
Pre-LN54.9444.6748.5337.32
LLaMA-style55.1445.1149.6640.91
QK-Norm56.3046.2451.3043.25
NormFormer43.1834.4935.5221.27
LayerScale56.3845.4850.8739.73
Appendix
Table 16: Tiny-ImageNet-200 from scratch with ViT-S, patch size 8, the DeiT recipe, and 200 epochs. Accuracy is reported as in Table 15 , and the noise, contrast, and input-scale columns average over the corresponding shift grids.
Figure 9: Same Burgers flow, different numerical units: qualitative field recovery at seed 0 after 20,000 steps. Columns use κ∈{10−2,1,102}\kappa\in\{10^{-2},1,10^{2}\} and are converted back to the same physical field u^/κ\hat{u}/\kappa . (a, b, c, e, f, g) Recovered field over space and time, with the recovered field’s relative L2L_{2} error above each panel, and black contours show the Cole–Hopf solution. (d, h) The same fields sliced at t=0.5t=0.5 against the exact solution (grey). Equal weighting in the top row changes the recovered field off-reference, whereas the NDA-typed head and objective in the bottom row recover consistent fields. Table 17 gives five-seed results.
problemmethodκ=10−2\kappa=10^{-2}κ=1\kappa=1κ=102\kappa=10^{2}worst
Burgersequal0.8480.0140.3370.848
manual λsup=100\lambda_{\mathrm{sup}}=1000.4910.0810.4310.491
uncertainty0.5200.0060.6180.618
LRA0.8120.0060.2300.812
NTK balance0.5600.2140.5500.560
NDA-typed (ours)0.0080.0090.0080.009
Appendix
Table 17: PINN relative L2L_{2} error of the recovered physical field, averaged over five seeds, when the data’s unit system is rescaled by κ\kappa . The problem is physically identical across columns, and only the units change. The last column gives the worst case over κ\kappa .
More in this topic

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 544 papers

Explore similar work

CardsList

  1. Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

    Apr 21, 2026Weijie Zhao, Mingquan Liu, Bolun Wang +4TransformerLanguage Model Scaling Laws

    Luxi Tech, Shenzhen, China · The School of Computer Science, China University of Geosciences, 388 Lumo Road, Wuhan, 430074, Hubei, China.

  2. Review Residuals: Update-Conditioned Residual Gating for Transformers

    Jun 30, 2026Kyle KramerTransformerResidual Learning

    NeraTech LLC

  3. Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

    Jul 27, 2026Jinhao Zhang, Zeyu Liu, Zicheng Yan +4Neural Network GeneralizationResidual Learning

    Beijing University of Posts and Telecommunications · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China +1