Deterministic, Reproducible Inference in Exact Integer Arithmetic, and Verifiable Distributed Inference as Its Consequence
Organizations: Pelloran Inc.
Abstract
Optimized floating-point inference is not reproducible across environments: the same model on different hardware, compilers or thread counts can produce different bits. We present tritify, an open-source runtime that executes the forward pass of real models -- decoder-only language models, Mixture-of-Experts decoders, and the Segment Anything 2.1 image/video stack -- with the entire run-time path in exact integer arithmetic; floating point is confined to offline quantization and the float reference used for verification. We make three separate claims and verify each. (1) Cross-hardware reproducibility: the integer path produces bit-identical output run to run, across CPU instruction-set architectures, and across CPU/GPU, carried by a reproducing SHA-256 of the output. (2) Decision fidelity: the integer path reproduces the float reference's decision on the models and inputs tested, with the numeric exceptions measured; per-layer fidelity through sixteen layers of OLMoE-1B-7B and of a 106-billion-parameter MoE shows no fidelity cliff. (3) Location independence: a model's computation can be split across instruction sets, CPU and GPU, and networked machines with zero effect on the output bits. GLM-4.5-Air (106,852,251,264 parameters, 427 GB of integer codes) generated its answer across three machines, none of which can hold its integer codes in RAM, bit-identically to a single-host reference. The consequence is verifiable distributed inference: an untrusted server or a fleet of devices runs the model while an open scalar oracle re-derives the exact bytes; a guarantee against a dishonest server is future work. We do not claim a per-device speed-up, bit-equality of floating-valued intermediates, or confidentiality. Code: https://github.com/pelloran-inc/tritify
Figures & tables
| Operation | Ref / metric | Probe / domain | Gate tol | M / D |
|---|---|---|---|---|
| Fixed-point requant, full-width mantissa | f64 twin, max abs | integer scale ratios | exact | M ( ) |
| Fixed-point requant, 8-bit mantissa | f64 twin, max abs | integer scale ratios | ulp | M |
| lookup, 20-bit index | f64 , max abs | table domain | M ( ) | |
| Inverse-RMS (integer isqrt ) | f64 , max rel | pts | M ( ) | |
| SAM 2.1 mask logits vs. float twin | max abs | base-plus image | diag. | M ( ) |
| Qwen2.5-0.5B logits vs. float | max abs | 512-token probe | diag. | M ( ) |
| Subject | Setup | Claim 1 (reproducibility) | Claim 2 (vs. float) |
|---|---|---|---|
| Ternary GEMM demo | , ; Apple M3 Max (Metal) vs. CPU | bit-identical CPU GPU; SHA-256 reproduced | exact integer GEMM (no float reference needed) |
| Decoder LM | Qwen2.5-0.5B, seq 512; Apple M3 Max (ARM/NEON) vs. AMD EPYC 9454P (x86-64/AVX-512), byte-identical weights | logits SHA-256 byte-identical ARM x86-64 | greedy tokens match on tested prompts; argmax on the 512-token probe |
| SAM 2.1 image | hiera-base-plus , 613-tensor checkpoint; self-check, no PyTorch | deterministic; reproducing mask SHA-256 | selected mask bit-identical ( px); IoU vs. float twin |
| Measurement | Speedup | Bit-identity |
|---|---|---|
| SIMD vs. vectorizers-off, hot kernel ( , rows) | (M3/NEON); (EPYC/AVX-512) | same digest; cross-ISA checksum equal |
| SIMD threads: Qwen2.5-0.5B forward_int , seq 512 | ( ) | all cells: same logits hash |
| SIMD threads: SAM 2.1 base-plus , end-to-end wall | ( ) | all cells: same mask-logits hash |
| Measure (over 16 layers) | OLMoE-1B-7B | GLM-4.5-Air (first 16) |
|---|---|---|
| Residual cosine, median | ||
| Residual cosine, min (final layer) | ||
| Residual cosine, worst position (any layer) | (recovers to ) | |
| Final logits cosine, median | ||
| Router top- exact agreement | ||
| Largest layer-to-layer median cosine drop |
| Property | Value | Meaning |
|---|---|---|
| Model codes | B | exceeds every node’s memory in this run |
| Coordinator held | B | router-only; holds_full_model: false |
| Workers resident (sum) | B | codes resident after warm-up (measured) the assigned routed experts’ codes; on-demand build, whole share fit here |
| Node memory (each) | M3 · host B · EPYC GB | each GB: no node holds the full codes in RAM |
| Bit-identity | 564f495c… both ways | distributed single-host streamed reference, all 46 layers |
| Cost | round-trips, s total | 90 round-trips per decode position ( MoE ); over prompt positions tokens; total wall, not a decode rate |
LLM Inference
Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.
See all 319 papers