Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization

Yakov Pyotr Shkolnikov

cs.LGMar 30, 2026

Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization

Authors: Yakov Pyotr Shkolnikov

Organizations: Independent Researcher, Mountain View, CA, United States

Abstract

Deep learning training is non-deterministic: identical code with different random seeds produces models that agree on aggregate metrics but disagree on individual predictions, with per-class AUC swings exceeding 20 percentage points on rare clinical classes. We present a framework for verified bit-identical training that eliminates three sources of randomness: weight initialization (via structured orthogonal basis functions), batch ordering (via golden ratio scheduling), and non-deterministic GPU operations (via architecture selection and custom autograd). The pipeline produces MD5-verified identical trained weights across independent runs. On PTB-XL ECG rhythm classification, structured initialization significantly exceeds Kaiming across two architectures (n=20; Conformer p = 0.016, Baseline p < 0.001), reducing aggregate variance by 2-3x and reducing per-class variability on rare rhythms by up to 7.5x (TRIGU range: 4.1pp vs 30.9pp under Kaiming, independently confirmed by 3-fold CV). A four-basis comparison at n=20 shows all structured orthogonal bases produce equivalent performance (Friedman p=0.48), establishing that the contribution is deterministic structured initialization itself, not any particular basis function. Cross-domain validation on seven MedMNIST benchmarks (n=20, all p > 0.14) confirms no performance penalty on standard tasks; per-class analysis on imbalanced tasks (ChestMNIST, RetinaMNIST) shows the same variance reduction on rare classes observed in ECG. Cross-dataset evaluation on three external ECG databases confirms zero-shot generalization (>0.93 AFIB AUC).

Figures & tables

Figure 1: Per-class performance variability (Conformer, $n\!=\!20$ seeds). (a) Kaiming initialization: SARRH ( $n_{\text{pos}}\!=\!77$ ) ranges 4.2pp across seeds despite being moderately represented. SVARR ( $n_{\text{pos}}\!=\!14$ ) ranges 20.2pp; TRIGU ( $n_{\text{pos}}\!=\!2$ ) ranges 30.9pp. (b) Mixed-basis initialization: systematically tighter ranges. SARRH reduced from 4.2 to 2.5pp, SVARR from 20.2 to 11.5pp, TRIGU from 30.9 to 4.1pp ( $7.5{\times}$ ). Green stars show the fully deterministic golden ratio run (zero variance by construction); golden falls within or above the seeded range for all classes. Thick horizontal lines: mean; shaded regions: $\pm$ 1 std; thin whiskers: full min–max range. Test-set positive counts (fold 10): SR 1674, AFIB 152, STACH 82, SARRH 77, SBRAD 64, PACE 28, SVARR 14, BIGU 8, AFLT 7, SVTAC 3, PSVT 2, TRIGU 2.

Figure 2: ECG Conformer architecture (1.83M parameters). The network processes 12-lead ECG input (1000 samples at 100 Hz) through a convolutional stem (3 $\times$ Conv1d + AvgPool, outputting 160 channels), three bottleneck stages with progressive stride-2 downsampling (3, 4, and 23 blocks respectively), a stride-2 transition convolution, and three Conformer blocks. Each Conformer block follows a macaron structure: FFN 1 (feed-forward network, $\times$ 0.5 residual) $\to$ MHSA (multi-head self-attention, 4 heads, $d\!=\!40$ ) $\to$ depthwise convolution ( $k\!=\!5$ ) $\to$ FFN 2 ( $\times$ 0.5 residual) $\to$ LN (layer normalization), with residual connections around each sub-layer. GAP (global average pooling) and GMP (global max pooling) are concatenated (320-d) and projected through a BN ( $K\!+\!2$ information bottleneck, 14-d) before 12 independent sigmoid heads. Pill badges above each stage indicate the orthogonal basis used for weight initialization in mixed-basis mode. Color coding: light cyan = stem/transition layers, light blue = convolutional bottleneck stages, light orange = Conformer blocks, light teal = information bottleneck, light green = classification heads.

Comparison	$n$	$d$	Power	$n_{80\%}$	TOST
Conform. mixed vs Kaim.	20	0.82	$\sim$ 71%	25	$p<0.001$ ( $\delta\!=\!0.015$ )
Baseline struct. vs Kaim.	20	1.60	$>$ 99%	7	superiority ( $p<0.001$ )
CIFAR-100 struct. vs Kaim.	20	0.24	11% a	—	$p<0.001$ ( $\delta\!=\!0.5$ pp)

Table 1: Statistical power analysis for primary comparisons. Observed Cohen’s

d

computed from pooled standard deviation; power estimated for two-sided independent

t

-test at

\alpha\!=\!0.05

;

n_{80\%}

is the per-group sample size required for 80% power.

Component	Test AUC	Delta
DCT-only (fixup=0.01)	$0.948\pm 0.012$	baseline
+ mixed bases	$0.951\pm 0.006$	+0.003
+ sqrt class weight	$0.951\pm 0.006$	+0.003
+ mixed + cw (“Mixed”)	$0.961\pm 0.003$	+0.013
Kaiming (reference)	$0.958\pm 0.002$	—
Kaiming + cw	$0.960\pm 0.006$ a	+0.002

Table 2: Ablation of structured initialization framework components (Conformer architecture). “DCT-only” = single-basis DCT across all stages, no class weighting. “Mixed” = DCT/Hadamard/Hartley per stage. “cw” = sqrt class weighting. Standard deviations reflect batch-ordering variance (

n\!=\!5

batch seeds per configuration); the Kaiming value differs from Table 3 (

n\!=\!20

initialization seeds) because it measures a different source of variance.

Figure 3: Structured orthogonal initialization. (a) DCT-II basis matrix for a Conv1d(64, 64, 5) layer: all 64 cosine basis vectors spanning from DC ( $k\!=\!0$ , top) to near-Nyquist ( $k\!=\!63$ , bottom). Each row is one deterministic filter; the structured frequency progression replaces random Kaiming weights. Alternative bases (Hadamard, Hartley, sinusoidal) use analogous constructions. (b) Convergence comparison (Conformer, $n\!=\!20$ seeds): mean $\pm$ 1 std bands. Mixed-basis initialization maintains higher validation AUC with lower variance from epoch 20 onward. Test-set results in Table 3 .

Model	Init	Configuration	Test AUC	Params	Det.
Conformer	Mixed	20 batch seeds	$\mathbf{0.959\pm 0.005}$	1.83M	Init
Conformer	Mixed	golden ratio	0.966 a	1.83M	Full
Conformer	Kaiming	20 init seeds	$0.953\pm 0.010$	1.83M	No
Baseline CNN	DCT	20 batch seeds	$0.956\pm 0.004$	1.65M	Init
Baseline CNN	Kaiming	20 init seeds	$0.947\pm 0.007$	1.65M	No
xresnet1d101	Kaiming	5 init seeds	$0.947\pm 0.013$	9.0M	No

Table 3: Structured initialization across architectures (PTB-XL rhythm classification, 12-class multi-label, macro AUC). “Det.” indicates determinism level. “Mixed” = mixed-basis initialization (DCT/Hadamard/Hartley per stage); “DCT” = single-basis DCT initialization. Both use sqrt class weighting. MedMNIST cross-domain results in Table 6 ; CIFAR-100 in Appendix A .

Basis (single, all stages)	Test AUC (mean $\pm$ std)	vs. DCT $p$ (paired)
Hartley-only	$\mathbf{0.959\pm 0.006}$	$0.253$
Sinusoidal-only	$0.956\pm 0.005$	$0.940$
Hadamard-only	$0.955\pm 0.008$	$0.751$
DCT-only	$0.956\pm 0.010$	—
Kaiming+cw	$0.954\pm 0.012$ a	—

Table 4: Orthogonal basis comparison (Conformer, PTB-XL rhythm classification,

n{=}20

seeds each). Each row uses a single basis uniformly across all stages with sqrt class weighting. Paired

t

-tests (same seeds across bases). Friedman

\chi^{2}{=}2.46

p{=}0.48

: no significant difference between structured bases. All standard deviations use

n{-}1

(sample std). a Kaiming+cw:

n{=}18

converged of 20 launched; 2 seeds (2, 7) collapsed to

<

0.67 AUC (0% failure rate for all structured bases and Kaiming without cw).

Architecture	Init	AUC	TRIGU	SVARR	AFLT
Conformer	Mixed	$\mathbf{0.959}$	$\mathbf{0.980}$	$0.892$	$\mathbf{0.951}$
Conformer	Kaiming	$0.928$	$0.781$	$0.923$	$0.855$
Baseline	Mixed	$0.950$	$0.906$	$0.870$	$0.963$
Baseline	Kaiming	$0.940$	$0.891$	$0.814$	$0.941$

Table 5: Golden-seed

2{\times}2

design: architecture crossed with initialization, all using golden ratio batch ordering. “Mixed” = mixed-basis (DCT/Hadamard/Hartley per stage) with sqrt class weighting. Per-class AUC shown for the three rhythms with the largest initialization effect. Each entry is a single deterministic run (MD5-verified for structured init). Caution: TRIGU (

n\!=\!2

test positives), SVARR (

n\!=\!14

), and AFLT (

n\!=\!7

) have small test-set sizes; per-class AUC estimates carry substantial uncertainty.

Dataset	DCT AUC	Kaiming AUC	$\Delta$	$p$	$n_{\text{train}}$
BloodMNIST	$.9986\!\pm\!.0002$	$.9987\!\pm\!.0002$	$-.0001$	.65	11,959
OrganCMNIST	$.9943\!\pm\!.0006$	$.9945\!\pm\!.0004$	$-.0002$	.16	13,000
DermaMNIST	$.9186\!\pm\!.0062$	$.9192\!\pm\!.0048$	$-.0006$	.74	7,007
BreastMNIST	$.878\!\pm\!.032$	$.877\!\pm\!.030$	$+.001$	.96	546
RetinaMNIST	$.729\!\pm\!.016$	$.734\!\pm\!.014$	$-.005$	.26	400
ChestMNIST	$.756\!\pm\!.011$	$.752\!\pm\!.014$	$+.004$	.32	78,468

Table 6: MedMNIST cross-domain validation. ResNet-18, 200 epochs,

n{=}20

seeds per initialization. DCT = single-basis 2D-DCT (no mixed bases, no class weighting).

p

-values from Welch’s

t

-test. No dataset shows a statistically significant difference.

Figure 4: Per-class AUC variability on ChestMNIST (ResNet-18, single-basis 2D-DCT, $n\!=\!20$ seeds). (a) Kaiming initialization: Pneumothorax ( $n_{\text{pos}}\!=\!1{,}089$ ) ranges 0.130, Emphysema ( $n_{\text{pos}}\!=\!509$ ) ranges 0.141. (b) DCT initialization: systematically tighter ranges in 11 of 14 classes (mean range 0.056 vs 0.079). Pneumothorax tightens $3.7{\times}$ (0.035 vs 0.130), Emphysema $2.4{\times}$ . Green stars: DCT golden deterministic run. Thick horizontal lines: mean; shaded regions: $\pm$ 1 std; thin whiskers: full min–max range. Compare with Fig. 1 for the analogous ECG pattern.

Figure 5: Generalization validation. (a) Three-fold cross-validation macro AUC for four configurations; Conformer combined ( $0.955\pm 0.012$ ) reaches the highest mean with seeded batch ordering. (b) Cross-dataset AFIB detection AUC on three external databases (AFDB, CPSC2018, Chapman-Shaoxing): both architectures exceed 0.93 transfer AUC.

Configuration	F10	F9	F8	Mean $\pm$ Std
Conform. mixed (seeded)	.942	.961	.963	$.955\!\pm\!.012$
Conform. mixed (golden)	.908	.948	.968	$.941\!\pm\!.031$
Baseline mixed	.927	.953	.934	$.938\!\pm\!.013$
Baseline Kaiming	.923	.949	.941	$.938\!\pm\!.013$

Table 7: Cross-validation results (3 folds, sample std). Each fold uses a single batch seed; Table 3 multi-seed results average 20 batch orderings on fold 10. The golden fold 10 result (0.908) differs from Table 3 (0.966) because CV golden uses a different validation fold for model selection, and golden ordering’s data-dependence amplifies fold sensitivity (std=0.031 vs 0.012 for seeded).

Model	AFDB	CPSC2018	Chapman	Params
Baseline (xresnet)	0.952	0.979	0.991	1.65M
Conformer	0.932	0.979	0.984	1.83M

Table 8: Cross-dataset generalization (6-lead models trained on 6 PTB-XL limb leads: I, II, III, aVR, aVL, aVF). Each entry is a single golden deterministic checkpoint. All models trained on PTB-XL with mixed-basis initialization and evaluated zero-shot for AFIB detection on three external databases. For AFDB, 4 of 6 leads are derived from 2 channels via Einthoven’s law (Section 3.7 ). These 6-lead models are distinct from the 12-lead models used in all other experiments.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Init	Accuracy	Params
CIFAR-100	DCT 2D	$78.4\pm 0.3$ %	11.2M
CIFAR-100	Kaiming	$78.3\pm 0.4$ %	11.2M

Appendix

Table 9: CIFAR-100 image classification results (ResNet-18, 20 seeds). DCT = single-basis 2D-DCT initialization. TOST equivalence confirmed at

\delta\!=\!0.5

percentage points (

p<0.001

Basis	Clean AUC	BW drop	Muscle drop	EM drop
Hartley	$\mathbf{.959\pm.003}$	$\mathbf{.033\pm.012}$	$\mathbf{.035\pm.010}$	$\mathbf{.057\pm.019}$
Hadamard	$.957\pm.008$	$.041\pm.019$	$.046\pm.023$	$.070\pm.022$
Sinusoidal	$.951\pm.006$	$.050\pm.011$	$.063\pm.021$	$.064\pm.019$
DCT	$.951\pm.005$	$.054\pm.009$	$.063\pm.017$	$.091\pm.017$

Appendix

Table 10: Noise robustness by orthogonal basis (Conformer, 12-lead PTB-XL,

n{=}5

seeds per basis, 0 dB SNR). Each column shows AUC drop from clean at the harshest noise level. Friedman tests: baseline wander

p{=}0.18

, muscle artifact

p{=}0.18

, electrode motion

p{=}0.069

; none reach significance at

\alpha{=}0.05

, though the electrode motion ordering approaches significance. Noise sources from the MIT-BIH Noise Stress Test Database [ 53 ] .

Method	AUC	Seed-free?	Notes
Seeded shuffle (control)	0.955	No	Upper bound
Class guaranteed	0.947	Yes	Highest seed-free; needs calibration
Golden ratio	0.941	Yes	Selected: parameter-free
Sobol sequence	0.939	Yes	Quasi-random coverage
Feature-space diverse	0.883	Yes	Rotating projection
Stratified quantile	0.579	Yes	Loss-based, class-correlated

Appendix

Table 11: Seed-free batch ordering strategies (Conformer, mixed-basis init, PTB-XL fold 10, single run each). All seed-free methods produce deterministic batch sequences without any random seed. Golden ratio was selected for simplicity (no hyperparameters, no class-label dependence) despite lower AUC than class-guaranteed ordering.

Figure 6: Theoretical analysis of DCT vs. Kaiming initialization. (a) Energy capture ratio for band-limited signals ( $N\!=\!64$ ): DCT with full rank preserves energy exactly; Kaiming has correct expectation but high variance. (b) Condition number: DCT is always 1 (optimal gradient flow); Kaiming grows with matrix dimension $N$ . (c) Mutual information $I(\mathbf{x};\,\mathbf{Wx})$ for a dimensionality-reducing layer ( $M\!=\!N/4$ , $N\!=\!64$ , SNR = 10 dB) with Toeplitz-correlated input: for correlated signals ( $\rho\gtrsim 0.3$ ), DCT preserves more mutual information than Kaiming by selecting the most informative spectral components; at low correlation, Kaiming’s higher per-filter variance (fan-in scaling) compensates for its random projection directions.

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 136 papers

OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network Verification LLM Auditing
Gensyn
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
Feb 19, 2026Akira Sakai, Yuma IchikawaModel Compression Neural Network Compression
Fujitsu Limited · Tokai University · Riken Center for AIP
The Order Is The Message
Mar 24, 2026Jordan LeDouxData-Efficient Learning Training Data Curation
Independent Researcher

Sep 15, 2026J/K move · Enter open · S save

OPEN-1B: A Fully Auditable Training Run

John Donaghy, Brian Wilcox, Oğuzhan Ersoy, Shikhar Rastogi+5

Gensyn

Neural Network Verification LLM Auditing Distributed Training ML Reproducibility

Open paper AskPDF arXiv

Open-source language models have a reproducibility problem. Despite releasing weights, training data, and recipes, none of them are provably reproducible due to the non-associativity of floating-point arithmetic. Deep learning frameworks often offer a deterministic execution mode, allowing reproducible operations on the same machines. Unfortunately, this determinism does not carry across hardware such that a user can verify that a released checkpoint was actually produced using the declared training recipe. This leaves room for undisclosed data, injected biases, or backdoors that existing techniques such as proof-of-learning or proof-of-training-data cannot rule out. We introduce a new tier of model transparency, fully auditable, in which every operation on every data sample during training is independently reproducible on heterogeneous commodity hardware with bitwise certainty. By imposing a definite order on the sources of training nondeterminism, GPU kernel reductions, data batch ordering across a data-parallel cluster, and inter/intra-node collective communication, we make it possible to replay any individual step of a large, distributed training run on a single piece of commodity hardware and check it against the published trajectory. Because replaying an entire run on one machine is infeasible, we support this with a collective verification scheme in which many independent auditors each certify individual steps, together covering the whole run. We release Open-1B, a model trained under this regime, together with its full pretraining dataset, every intermediate checkpoint, the training codebase, and the audit harness needed to reproduce and verify any step of its training.

Abstract

ML Reproducibility

Explore similar work

OPEN-1B: A Fully Auditable Training Run

Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

The Order Is The Message