Testing Processor Contracts in Speculative Decoding Engines
Organizations: Independent Researcher
Abstract
Speculative sampling can accelerate decoding without changing the target distribution. Its standard guarantee assumes that logits processors, which enforce constraints or modify token scores, receive the appropriate token prefix and state at each verification position. We formalize this processor-prefix contract and present BADC (branch-aware differential conformance testing). BADC combines enumerated references on tiny random models, checks of processor inputs and state, and simultaneous confidence bounds on output-event probabilities. Preregistered, source-informed experiments on selected Transformers and llama-cpp-python releases confirm two causes: processors read unverified drafts or advance state without rollback. For one pilot-selected prompt on a trained checkpoint, fresh samples give a 60.26-percentage-point lower bound on the difference in the probability of a predefined first-token event between ordinary and prompt-lookup decoding, with nominal simultaneous 95% coverage under within-arm i.i.d. sampling. Controlled comparisons establish no cause-recovery advantage of directed selection over random selection, while permutation testing detects each tested family at least as often as BADC. These findings connect processor-interface failures to observable decoding differences without establishing diagnostic superiority or workload prevalence.
Figures & tables
| Check | Supported interpretation |
|---|---|
| Detection | A contract witness or a confidence lower bound on the arms’ total-variation distance. Non-rejection is not conformance. |
| Compatibility | Consistency with one supplied mechanism law under the registered rule; alternatives are not excluded. |
| Repair | Fresh recurrence and upper bounds on frozen events, not a global conformance guarantee. |
| Exercise | Hooked draft proposals. The LCP repair acceptance counters are inexact; they do not establish precise acceptance or speedup. |
| Config | Feature probed | Role | Decision | Witness | Eligible | Hypothesis | Proposed | |
|---|---|---|---|---|---|---|---|---|
| HF12-P2 | plain sampling (5.12.0) | pos. control | FAIL S | 623 | 0.0025 | yes | – | 925,878 |
| HF17-N0 | plain sampling ( ) | neg. control | NR | 0 | 0.0001 | no | – | 927,150 |
| HF17-P1 | stop_strings =[”cd”] | pos. control | FAIL S | 1,144 | 0.0052 | yes | – | 925,554 |
| LCP-N0-19/35 | plain, generate() | neg. control | NR | 0 | 0.0001 | no | – | 1,067,543/1,067,406 |
| LCP-N1-19/35 | plain, create_completion() | neg. control | NR | 0 | 0.0001 | no | – | 985,359/984,606 |
| HF17-3 | , top- 3, top- 0.9, rep. 1.3 | probe | INC. | 0 | 0.0001 | no | – | 0 |
| Config | Build | Frozen event | Recurrence | Repair gap | Witness UB | Confirmed | [interval] |
|---|---|---|---|---|---|---|---|
| HF17-6 | 5.17.0 | 0.0195 | 0.0032 | yes | [ , ] | ||
| LCP-2-19 | 0.3.19 | 0.1549 | 0.0020 | yes | [ , ] | ||
| LCP-2-35 | 0.3.35 | 0.1548 | 0.0023 | yes | [ , ] |
| Evidence | Claim and limit |
|---|---|
| Historical repairs | Selected tiny-model probes; fresh-draw bounds on frozen repair events. |
| Trained scores | Four-prompt pilot and eight fixed matrix states; conditional-score/history checks, not returned-law inference. |
| Returned event | One selected prompt/checkpoint, 1,024 requests/arm; nominal event-gap bound under within-arm i.i.d. sampling. |
| Second host | Same collector, prompt, event and seeds; reproducibility without pooled . |
| (a) Selection, events | stateful processors | processor buffer | causes (of 20) |
|---|---|---|---|
| directed, with | 10 | 9–10 | 19–20 |
| directed, without | 10 | 9–10 | 19–20 |
| random, with | 10 | 10 | 20 |
| random, without | 9 | 10 | 19 |
| directed | random | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| (a) Family | path | exp. | Badc | C | recurred | valid | exp. | Badc | C |
| allowed tokens | V1 | 5 | 5 | 5 | 5 of 5 | 4 | 4 | 4 | 4 |
| bad words | V1 | 4 | 4 | 4 | 3 of 4 (1 missing) | 5 | 4 | 4 | 4 |
| minimum tokens | V1 | 5 | 5 | 5 | 5 of 5 | 5 | 5 | 5 | 5 |
| stop strings | V1 | 5 | 5 | 5 | 5 of 5 | 5 | 3 | 3 | 3 |
| logit bias | V1 | 5 | 5 | 5 | 5 of 5 | 5 | 5 | 5 | 5 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Marginal | Faulty/off | Intervention/off |
|---|---|---|
| HF first token | 1.2971 | 0.1479 |
| HF length | 0.8916 | 0.0441 |
| LCP first token | 0.2686 | 0.2844 |
| Fixed repetitive prompt | Initial conditional TV (%) |
|---|---|
| Paris factual sentence | 65.8317 |
| Numerical sentence | 4.1911 |
| Unusual lexical sentence | 3.8606 |
| Short word sequence | 45.3831 |
| Arm | Count | Rate | Interval |
|---|---|---|---|
| Ordinary | 823 | 80.3711 | [77.2287, 83.2620] |
| Native lookup | 145 | 14.1602 | [11.6540, 16.9638] |
| History control | 833 | 81.3477 | [78.2583, 84.1774] |
| Prompt | Plain E | Repetition E | Native CFG E/W | Prefix CFG E/W | Native CFG initial TV |
|---|---|---|---|---|---|
| P01 | 2 | 2 | 2/2 | 2/0 | 0.0224984 |
| P02 | 2 | 2 | 2/2 | 2/0 | 0.0347482 |
| P03 | 2 | 2 | 2/2 | 2/0 | 0.136422 |
| P04 | 2 | 2 | 2/2 | 2/0 | 0.00240589 |
| P05 | 0 | 1 | 0/0 | 1/0 | 0 |
| P06 | 2 | 2 | 2/2 | 2/0 | 0.380219 |
| Arm | Event | First | Full |
|---|---|---|---|
| Ordinary | 823 | 1,024 | 1,024 |
| Native lookup | 145 | 1,024 | 998 |
| Prefix control | 833 | 1,024 | 939 |
| Omitted class | CP-set | TV-nearest | |
|---|---|---|---|
| HF-Bplus | 256 | 128/0/0 | 0/114/14 |
| HF-Bplus | 2,048 | 128/0/0 | 128/0/0 |
| HF-Bplus | 16,384 | 128/0/0 | 128/0/0 |
| HF-Bminus | 256 | 1/48/79 | 0/124/4 |
| HF-Bminus | 2,048 | 128/0/0 | 128/0/0 |
| HF-Bminus | 16,384 | 128/0/0 | 128/0/0 |
| Generating class | CP-set | TV-nearest |
|---|---|---|
| null | 46/0/82 | 121/0/7 |
| HF-Bplus | 128/0/0 | 128/0/0 |
| HF-Bminus | 0/0/128 | 128/0/0 |
| HF-D | 128/0/0 | 128/0/0 |
| HF-Kx | 128/0/0 | 122/6/0 |
| HF-F | 122/0/6 | 122/0/6 |
Speculative Decoding
Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.
See all 343 papers