Language model APIs can expose token-specific biases while withholding logits and token probabilities. Existing bias-based extraction recovers relative logits, leaving a common additive component undetermined in ideal arithmetic. We identify a numerical side channel in floating-point bias addition that exposes information about this component through token identities alone. The channel distinguishes score vectors with identical complete unperturbed log probabilities. Our method exploits changes in representable spacing at binary exponent boundaries. Under the stated binary32 arithmetic assumptions and a known reachable score interval, it recovers a logit pair and its unknown tie-breaking priority exactly in at most 52 token queries. Given complete relative logits, one recovered anchor determines the original logit vector. We evaluate recovery across 11 public checkpoints from seven model families and extend the local method with adaptive interval search and anchor selection. In end-to-end experiments with vLLM, SGLang, and llama.cpp, adaptive recovery succeeds on all 128 Qwen3-0.6B inputs per system, including 64 held-out inputs. A separate token-only experiment extracts a pretrained small model's complete output-head representation up to an invertible hidden-basis transformation. We also establish a sufficient condition for suppressing common-shift disclosure by subtracting the maximum before bias addition. In controlled experiments, this mitigation eliminates all observed distinctions between common-shift variants and preserves token decisions under the tested random biases.
Figures & tables
Fig. 1: Overview of the numerical channel and recovery. (a) The observer chooses token biases b for fixed input x and observes the selected token; RN32 denotes binary32 rounding. (b) At the positive boundary P=2e , spacing increases from Δ to 2Δ . At the illustrated point below P , the increment is exact. At P , the exact sum P+Δ lies halfway between P and P+2Δ and rounds to P under ties-to-even. (c) Gap and priority recovery followed by boundary probes recovers the score pair under the stated arithmetic and range assumptions. (d) Subtracting the maximum before bias suppresses common-shift disclosure when stored variants yield identical centered vectors.
Fig. 2: Token responses for q=(30,28) and q′=q−811 . The anchor bias is 2 ; the target bias is 4+u2−21 for integer offsets u . The vectors have identical unperturbed probabilities. Shading marks offsets with different winners. The anchor wins ties between rounded scores; connecting lines guide the eye.
Fig. 3: Eliminating the periodic response. (a) Full-grid probes are periodic for p≤a and constant for p>a . (b) The recovered residue class (open markers) gives a monotone comparison; gray points are excluded. The change at p=a corresponds to floating-point spacing changing from 2Δ to Δ . Lines connect discrete queries. The horizontal origin uses the unknown a for illustration.
Generation
Cached multi-window search
Model
H
Boundary
Exact
Calls
Boundaries
Fixed
Selected
Pythia-14M
100
16
64
3,284
16
64
64
Pythia-160M
100
–
0
1,600
–
0
0
Pythia-1.4B
100
16
64
3,269
16
64
64
GPT-2 Medium
100
−64
54
3,136
−64,−32
62
64
GPT-Neo-125M
100
−1
13
2,363
−2,−1,−0.5
19
64
TABLE I: FP32 recovery with a calibrated interval and adaptive anchor selection.
Generation
Cached
Model
H
Exact
Generic
Known
Selected
Pythia-160M
100
0
–
–
0
Pythia-160M
256
32
48
13
32
OPT-1.3B
100
30
50
14–15
32
BLOOM-560M
100
28
46–48
11–13
28
BLOOM-560M
160
NR
NR
NR
32
TABLE II: BF16 recovery with FP32 bias processing.
Single window
Adaptive Qwen3
Framework
Qwen3
SmolLM2
Dev.
Test
Calls
vLLM
37
64
64
64
119.6
SGLang
36
64
64
64
120.4
llama.cpp
36
64
64
64
115.4
TABLE III: End-to-end recovery in LLM serving systems.
Input set
Extraction/test
Exact anchors
Relative head error
Mean RMSE
Token decisions
HTTP batches
1
96/16
96/96
3.96×10−6
3.11×10−7
1,370,786
45,873
2
192/32
192/192
1.19×10−6
4.59×10−7
2,740,717
91,584
3
96/16
96/96
3.35×10−6
7.21×10−7
1,372,446
46,928
TABLE IV: Token-only extraction of TinyStories-260K’s complete 512×64 output head and held-out mean prediction.
Fig. 4: Token-only output-head extraction from TinyStories-260K across three independently generated input sets. (a) Empirical distributions of absolute entry errors against the original W , after the same basis alignment. Solid curves include the recovered shared component; dashed curves omit it. The relative branch’s own target is CW . (b) Output-head relative error as the extraction input count increases, with each set’s test inputs fixed. Crosses mark inferred ranks below 64; set 2 is unresolved at 32–64 inputs and has no corresponding error points. Lines connect measured counts.
Fig. 5: Exact recovery and cost under independent response flips. Single uses one response per comparison; Majority-9 uses nine-response voting; LR and CI use sequential likelihood-ratio and confidence-interval tests, respectively. Each point aggregates 504 fixed in-range pairs and five noise seeds. Cost counts responses after pair discovery. The inset enlarges LR and CI near 100%; the cost axis is logarithmic. Lines connect measured conditions.
Rule
Exact recovery
Mean responses
Single
17/2,520
47.8
Majority-9
2,411/2,520
437.4
LR
2,502/2,520
242.7
CI
2,520/2,520
2,944.2
TABLE V: Recovery at 10% independent response flips.
Fig. 6: Cross-shift winner disagreements under three processing orders. Pre and Post apply max subtraction before and after bias addition, respectively. Rates use 38,968 comparisons: 2,578 disagreements for Original and Post, and 0 for Pre. The experiment uses fixed-grid decoder groups; unperturbed probabilities and argmax decisions remain bitwise unchanged in all 160 conditions.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.