Watermarkable Multi-Draft Speculative Sampling via Poisson Processes

Yanxiao Liu; Sicheng Wan; Zhan Gao; Deniz Gündüz

cs.CRSep 18, 2026

Watermarkable Multi-Draft Speculative Sampling via Poisson Processes

Authors: Yanxiao Liu, Sicheng Wan, Zhan Gao, Deniz Gündüz

Organizations: Imperial College London · University of Washington

Abstract

Large language models (LLMs) have achieved state-of-the-art performance across a wide range of tasks, motivating two important aspects of deployment: inference efficiency and output provenance, which can be tackled by speculative sampling and watermarking, respectively. However, recent works have shown that combining these two goals is highly nontrivial and can be potentially impossible. In this work, we develop a novel multi-draft speculative sampling algorithm based on Poisson processes that improves the frontier of this fundamental trade-off. The proposed algorithm has strong sampling efficiency on its own and, more interestingly, is naturally watermarkable: we can embed an unbiased watermark without degrading speculative acceptance. Moreover, our algorithm is based on an exact list-coupling-without-communication scheme, which yields a drafter invariance property that benefits both sampling and watermarking. It is the first multi-draft, drafter-invariant speculative sampling scheme that maintains both watermark strength and sampling efficiency, and we experimentally verify its strong performance in both aspects.

Figures & tables

Table 1

Figure 1: The trade-off between watermark and efficiency on existing methods ( Hu and Huang, 2024 ) : Vanilla Speculative Sampling (VSpS), Maintain Watermark Strength (MWS), and Maintain Sampling Efficiency (MSE), and our algorithms (PFR, and multi-draft MPFR that is presented in Section 5 ). For both PFR (lower-left) and MPFR (upper), we embed watermark (measured by Average Negative Log P-value Per Token (ANLPPT)) without degrading the Average Accepted Tokens Per Step (AATPS). This figure uses Qwen2.5-7B-Instruct and Qwen2.5-0.5B-Instruct as the target and draft, respectively, on dataset cnn_dailymail . Experiments on more models and datasets are shown in Appendix D .

		Qwen-CNN/ DM		Qwen- ELI5		Llama-CNN/ DM		Llama- ELI5
Decoder	$B$	AATPS	TR	AATPS	TR	AATPS	TR	AATPS	TR
MPFR	2	$2.797$	$\mathbf{25.11}$	$\mathbf{2.563}$	$\mathbf{23.57}$	$\mathbf{3.459}$	$\mathbf{37.44}$	$\mathbf{3.103}$	$\mathbf{34.29}$
MPFR	4	$\mathbf{3.123}$	$\mathbf{27.50}$	$\mathbf{2.918}$	$\mathbf{26.46}$	$\mathbf{3.784}$	$\mathbf{40.20}$	$\mathbf{3.455}$	$\mathbf{37.79}$
MPFR	6	$\mathbf{3.294}$	$\mathbf{28.50}$	$\mathbf{3.109}$	$\mathbf{27.81}$	$\mathbf{3.935}$	$\mathbf{41.71}$	$\mathbf{3.635}$	$\mathbf{39.59}$
MPFR	8	$\mathbf{3.409}$	$\mathbf{29.13}$	$\mathbf{3.235}$	$\mathbf{28.73}$	$\mathbf{4.034}$	$\mathbf{42.04}$	$\mathbf{3.749}$	$\mathbf{39.76}$
GLS	2	$\mathbf{2.803}$	$24.78$	$2.523$	$22.73$	$3.431$	$36.79$	$3.080$	$33.72$

Table 1: Multi-draft results on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct on CNN/ DailyMail and ELI5 ,

L=4

. We report the average accepted tokens per step (AATPS) and token rate (TR). The better result between MPFR and GLS for each setting with the same

B

is in bold. Each cell is computed over the

1000

prompts and

3

different seeds. See more detailed data in Appendix D.4 .

Figure 2: Detection and robustness to drafter substitution. (a) TPR@1%FPR versus detection token budget $T_{\mathrm{eval}}$ at the default drafter. PFR and MPFR match the strong-watermark references Basic-UWM and MWS , and outperform MSE / MSE-Pseudo . (b) TPR@1%FPR under drafter substitution: the target, watermark key, and prompts are fixed and only the drafter is swapped. PFR drifts by $<\!1$ point, whereas MSE drifts by $12$ – $22$ points and MSE-Pseudo by $6$ – $11$ points.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

(S_{i_{1}},i_{1},Z_{i_{1}}),\ldots,(S_{i_{B}},i_{B},Z_{i_{B}})

Appendix

Algorithm 5 Finite implementation of the

B

-sample mapped PFR

(S_{(1)},U_{(1)},J_{(1)}),\ldots,(S_{(B)},U_{(B)},J_{(B)})

Appendix

Algorithm 6 Direct finite-support implementation of

B

-sample MPFR

(X_{1}(c),\ldots,X_{\nu(c)}(c))\leftarrow\textsc{MPFR}(Q_{k}(\cdot|c),\Pi_{\mathsf{k}}(c),\nu(c)).

Appendix

Algorithm 7 Batched-target direct-top-

k

MPFR speculative sampling

Figure 3: Single-draft (AATPS, ANLPPT-U) frontier on Qwen2.5-7B-Instruct / Qwen2.5-0.5B-Instruct $\times$ ELI5 . PFR dominates MWS / MSE simultaneously on both axes; attaching the watermark does not decrease AATPS.

Figure 4: Single-draft (AATPS, ANLPPT-U) frontier on Vicuna-7B-v1.5 / Vicuna-68m $\times$ CNN/DailyMail . PFR dominates MWS / MSE simultaneously on both axes; attaching the watermark does not decrease AATPS.

Figure 5: Single-draft (AATPS, ANLPPT-U) frontier on Vicuna-7B-v1.5 / Vicuna-68m $\times$ ELI5 . PFR dominates MWS / MSE simultaneously on both axes; attaching the watermark does not decrease AATPS.

$L$	Method	AATPS	Token rate	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
1	Basic-UWM	$1.000\pm 0.000$	$33.353\pm 2.815$	$0.048\pm 0.035$	$0.035\pm 0.029$	$0.051\pm 0.036$	$0.473\pm 0.134$
	VSpS	$1.641\pm 0.070$	$33.229\pm 1.944$	$0.003\pm 0.005$	$0.003\pm 0.006$	$0.003\pm 0.006$	$0.477\pm 0.136$
	PFR-NoWM	$1.623\pm 0.075$	$32.147\pm 2.161$	$0.002\pm 0.006$	$0.003\pm 0.006$	$0.002\pm 0.006$	$0.474\pm 0.140$
	PFR	$1.625\pm 0.072$	$33.321\pm 1.706$	$0.051\pm 0.036$	$0.036\pm 0.030$	$0.053\pm 0.038$	$0.470\pm 0.143$
	MSE	$1.642\pm 0.070$	$20.578\pm 2.645$	$0.023\pm 0.021$	$0.016\pm 0.017$	$0.025\pm 0.023$	$0.472\pm 0.136$
	MWS	$1.621\pm 0.072$	$20.136\pm 2.952$	$0.049\pm 0.035$	$0.034\pm 0.029$	$0.052\pm 0.036$	$0.475\pm 0.133$

Appendix

Table 2: Single-draft results on Qwen2.5-7B-Instruct / Qwen2.5-0.5B-Instruct

\times

CNN/DailyMail . Each cell:

\mathrm{mean}\pm\mathrm{std}

across

1000

prompts.

$L$	Method	AATPS	Token rate	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
1	Basic-UWM	$1.000\pm 0.000$	$26.132\pm 1.924$	$0.100\pm 0.045$	$0.072\pm 0.039$	$0.104\pm 0.046$	$0.696\pm 0.141$
	VSpS	$1.593\pm 0.060$	$32.090\pm 1.889$	$0.002\pm 0.005$	$0.002\pm 0.005$	$0.002\pm 0.005$	$0.701\pm 0.133$
	PFR-NoWM	$1.567\pm 0.061$	$33.046\pm 1.405$	$0.002\pm 0.004$	$0.002\pm 0.004$	$0.002\pm 0.004$	$0.703\pm 0.139$
	PFR	$1.570\pm 0.062$	$33.213\pm 1.487$	$0.104\pm 0.048$	$0.075\pm 0.039$	$0.107\pm 0.049$	$0.699\pm 0.134$
	MSE	$1.595\pm 0.060$	$18.218\pm 2.022$	$0.048\pm 0.030$	$0.032\pm 0.024$	$0.053\pm 0.032$	$0.704\pm 0.135$
	MWS	$1.565\pm 0.063$	$20.177\pm 2.688$	$0.102\pm 0.047$	$0.075\pm 0.040$	$0.106\pm 0.048$	$0.700\pm 0.137$

Appendix

Table 3: Single-draft results on Qwen2.5-7B-Instruct / Qwen2.5-0.5B-Instruct

\times

ELI5 . Each cell:

\mathrm{mean}\pm\mathrm{std}

across

1000

prompts.

$L$	Method	AATPS	Token rate	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
1	Basic-UWM	$1.000\pm 0.000$	$37.058\pm 0.848$	$0.021\pm 0.024$	$0.016\pm 0.019$	$0.022\pm 0.025$	$0.274\pm 0.190$
	VSpS	$1.557\pm 0.086$	$48.890\pm 3.439$	$0.002\pm 0.006$	$0.003\pm 0.006$	$0.002\pm 0.006$	$0.275\pm 0.167$
	PFR-NoWM	$1.544\pm 0.088$	$37.027\pm 5.064$	$0.002\pm 0.006$	$0.003\pm 0.006$	$0.002\pm 0.005$	$0.269\pm 0.121$
	PFR	$1.550\pm 0.090$	$38.256\pm 6.092$	$0.022\pm 0.024$	$0.017\pm 0.019$	$0.023\pm 0.025$	$0.272\pm 0.128$
	MSE	$1.557\pm 0.088$	$45.513\pm 4.490$	$0.009\pm 0.013$	$0.007\pm 0.011$	$0.010\pm 0.015$	$0.269\pm 0.123$
	MWS	$1.547\pm 0.091$	$30.814\pm 4.078$	$0.022\pm 0.024$	$0.017\pm 0.020$	$0.023\pm 0.026$	$0.276\pm 0.192$

Appendix

Table 4: Single-draft results on Vicuna-7B-v1.5 / Vicuna-68m

\times

CNN/DailyMail . Each cell:

\mathrm{mean}\pm\mathrm{std}

across

1000

prompts.

$L$	Method	AATPS	Token rate	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
1	Basic-UWM	$1.000\pm 0.000$	$38.536\pm 1.896$	$0.060\pm 0.040$	$0.044\pm 0.033$	$0.062\pm 0.042$	$0.507\pm 0.152$
	VSpS	$1.410\pm 0.063$	$46.306\pm 3.178$	$0.002\pm 0.005$	$0.003\pm 0.005$	$0.002\pm 0.005$	$0.511\pm 0.150$
	PFR-NoWM	$1.399\pm 0.063$	$33.636\pm 4.757$	$0.003\pm 0.006$	$0.003\pm 0.007$	$0.002\pm 0.005$	$0.509\pm 0.152$
	PFR	$1.397\pm 0.065$	$35.231\pm 5.397$	$0.061\pm 0.043$	$0.045\pm 0.038$	$0.063\pm 0.042$	$0.513\pm 0.162$
	MSE	$1.415\pm 0.062$	$44.719\pm 3.192$	$0.020\pm 0.020$	$0.014\pm 0.017$	$0.022\pm 0.022$	$0.509\pm 0.147$
	MWS	$1.399\pm 0.064$	$26.309\pm 3.816$	$0.060\pm 0.041$	$0.044\pm 0.034$	$0.062\pm 0.042$	$0.507\pm 0.151$

Appendix

Table 5: Single-draft results on Vicuna-7B-v1.5 / Vicuna-68m

\times

ELI5 . Each cell:

\mathrm{mean}\pm\mathrm{std}

across

1000

prompts.

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	$2.797\pm 0.001$	$25.11\pm 0.22$	$0.051\pm 0.001$	$0.036\pm 0.001$	$0.053\pm 0.001$	$0.474\pm 0.002$
MPFR	4	$3.123\pm 0.002$	$27.50\pm 0.31$	$0.051\pm 0.001$	$0.036\pm 0.001$	$0.053\pm 0.001$	$0.474\pm 0.002$
MPFR	6	$3.294\pm 0.003$	$28.50\pm 0.49$	$0.050\pm 0.001$	$0.036\pm 0.001$	$0.053\pm 0.001$	$0.474\pm 0.001$
MPFR	8	$3.409\pm 0.002$	$29.13\pm 0.59$	$0.051\pm 0.002$	$0.036\pm 0.001$	$0.053\pm 0.002$	$0.474\pm 0.002$
Invariant	2	$2.803\pm 0.007$	$24.78\pm 0.14$	$0.002\pm 0.000$	$0.003\pm 0.000$	$0.002\pm 0.000$	$0.483\pm 0.005$
Invariant	4	$3.109\pm 0.008$	$27.12\pm 0.20$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.484\pm 0.003$

Appendix

Table 6: Multi-draft results on Qwen2.5-7B-Instruct

\times

CNN/DailyMail ,

L{=}4

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	$2.563\pm 0.007$	$23.57\pm 0.15$	$0.104\pm 0.001$	$0.075\pm 0.001$	$0.108\pm 0.001$	$0.705\pm 0.001$
MPFR	4	$2.918\pm 0.008$	$26.46\pm 0.25$	$0.104\pm 0.001$	$0.076\pm 0.001$	$0.108\pm 0.001$	$0.706\pm 0.000$
MPFR	6	$3.109\pm 0.005$	$27.81\pm 0.35$	$0.103\pm 0.001$	$0.075\pm 0.001$	$0.107\pm 0.001$	$0.705\pm 0.000$
MPFR	8	$3.235\pm 0.010$	$28.73\pm 0.49$	$0.104\pm 0.002$	$0.075\pm 0.002$	$0.108\pm 0.001$	$0.706\pm 0.001$
Invariant	2	$2.523\pm 0.004$	$22.73\pm 0.22$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.747\pm 0.005$
Invariant	4	$2.850\pm 0.010$	$25.60\pm 0.26$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.736\pm 0.002$

Appendix

Table 7: Multi-draft results on Qwen2.5-7B-Instruct

\times

ELI5 ,

L{=}4

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	$3.459\pm 0.015$	$37.44\pm 0.57$	$0.058\pm 0.001$	$0.041\pm 0.000$	$0.061\pm 0.001$	$0.514\pm 0.002$
MPFR	4	$3.784\pm 0.013$	$40.20\pm 0.28$	$0.058\pm 0.000$	$0.041\pm 0.000$	$0.062\pm 0.001$	$0.514\pm 0.003$
MPFR	6	$3.935\pm 0.005$	$41.71\pm 0.30$	$0.058\pm 0.000$	$0.041\pm 0.000$	$0.061\pm 0.001$	$0.514\pm 0.003$
MPFR	8	$4.034\pm 0.005$	$42.04\pm 0.81$	$0.058\pm 0.001$	$0.041\pm 0.000$	$0.061\pm 0.001$	$0.514\pm 0.003$
Invariant	2	$3.431\pm 0.005$	$36.79\pm 0.37$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.514\pm 0.003$
Invariant	4	$3.741\pm 0.007$	$39.31\pm 0.29$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.508\pm 0.001$

Appendix

Table 8: Multi-draft results on Llama-3.1-8B-Instruct

\times

CNN/ DailyMail ,

L=4

. Mean

\pm

std over 3 random seeds,

1000

prompts each. The drafter is Llama-3.2-1B-Instruct, the official distillation of the target sharing the same tokenizer.

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	$3.103\pm 0.006$	$34.29\pm 0.35$	$0.135\pm 0.001$	$0.099\pm 0.001$	$0.139\pm 0.001$	$0.811\pm 0.004$
MPFR	4	$3.455\pm 0.010$	$37.79\pm 0.38$	$0.135\pm 0.002$	$0.099\pm 0.001$	$0.139\pm 0.002$	$0.812\pm 0.003$
MPFR	6	$3.635\pm 0.001$	$39.59\pm 0.43$	$0.135\pm 0.002$	$0.099\pm 0.001$	$0.139\pm 0.002$	$0.812\pm 0.005$
MPFR	8	$3.749\pm 0.002$	$39.76\pm 0.80$	$0.135\pm 0.001$	$0.098\pm 0.000$	$0.139\pm 0.001$	$0.811\pm 0.004$
Invariant	2	$3.080\pm 0.011$	$33.72\pm 0.60$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.803\pm 0.003$
Invariant	4	$3.408\pm 0.006$	$37.11\pm 0.12$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.002\pm 0.000$	$0.794\pm 0.004$

Appendix

Table 9: Multi-draft results on Llama-3.1-8B-Instruct

\times

ELI5,

L=4

. Mean

\pm

std over 3 random seeds,

1000

prompts each.

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	2.799	23.18	0.052	0.037	0.055	0.475
MPFR	4	3.124	24.89	0.052	0.037	0.055	0.474
MPFR	6	3.298	25.39	0.052	0.037	0.055	0.476
MPFR	8	3.410	25.17	0.053	0.037	0.055	0.476
Invariant	2	2.804	25.03	0.003	0.003	0.002	0.487
Invariant	4	3.114	26.63	0.002	0.002	0.002	0.481

Appendix

Table 10: Multi-draft results on Qwen2.5-7B-Instruct

\times

CNN/DailyMail ,

L{=}4

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	2.563	22.38	0.104	0.076	0.108	0.704
MPFR	4	2.919	24.48	0.104	0.076	0.107	0.703
MPFR	6	3.111	25.54	0.105	0.076	0.108	0.706
MPFR	8	3.237	25.83	0.105	0.076	0.108	0.704
Invariant	2	2.519	23.18	0.002	0.002	0.002	0.750
Invariant	4	2.852	25.75	0.002	0.002	0.002	0.733

Appendix

Table 11: Multi-draft results on Qwen2.5-7B-Instruct

\times

ELI5 ,

L{=}4

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	2.422	52.10	0.022	0.017	0.022	0.270
MPFR	4	2.664	48.28	0.022	0.017	0.023	0.271
MPFR	6	2.803	44.44	0.022	0.017	0.023	0.270
MPFR	8	2.899	40.76	0.022	0.017	0.023	0.270
Invariant	2	2.478	58.15	0.002	0.002	0.002	0.287
Invariant	4	2.690	53.12	0.002	0.002	0.002	0.286

Appendix

Table 12: Multi-draft results on Vicuna-7B-v1.5

\times

CNN/DailyMail ,

L{=}4

Decoder	$B$	AATPS	TR	ANLPPT-U	ANLPPT-Li	ANLPPT-PL	LPPL
MPFR	2	1.889	45.89	0.058	0.043	0.061	0.504
MPFR	4	2.133	47.43	0.058	0.042	0.061	0.504
MPFR	6	2.278	47.42	0.058	0.042	0.062	0.503
MPFR	8	2.377	46.42	0.059	0.043	0.062	0.504
Invariant	2	1.884	50.29	0.002	0.002	0.002	0.555
Invariant	4	2.108	53.42	0.002	0.002	0.002	0.552

Appendix

Table 13: Multi-draft results on Vicuna-7B-v1.5

\times

ELI5 ,

L{=}4

Figure 6: Full detection across $(\textsc{target},\textsc{dataset})$ cells. TPR@1%FPR (Aaronson Gamma-tail) versus detection token budget $T_{\mathrm{eval}}$ at the default drafter $D_{0}$ , for the seven schemes plotted in Figure 2 (a). Each panel is one cell of the main $2{\times}2$ result: target $\in$ {Qwen2.5-7B-Instruct, Vicuna-7B-v1.5} $\times$ dataset $\in$ { CNN/DailyMail , ELI5 }. PFR / MPFR (ours) match the strong-watermark ceiling ( Basic-UWM , MWS ) in every cell, while MSE and MSE-Pseudo are uniformly weaker. The empirical $H_{0}$ (no-watermark) floor sits near the nominal $1\%$ FPR, calibrating the test.

	Single-draft baselines							Multi-draft ( $B{=}4$ )
Metric	VSpS	PFR-NoWM	Basic-UWM	MSE	MWS	MSE-Pseudo	PFR	Invariant	MPFR
LPPL $\downarrow$	0.474	0.469	0.472	0.470	0.475	0.477	0.465	0.484	0.474
ROUGE-L $\uparrow$	0.219	0.222	0.223	0.221	0.223	0.219	0.223	0.222	0.222
AATPS $\uparrow$	2.51	2.46	1.00	2.53	2.43	2.52	2.46	3.12	3.13

Appendix

Table 14: Output quality. LPPL

\downarrow

under target. ROUGE-L

\uparrow

vs. gold highlights . AATPS

\uparrow

accepted target tokens per step.

Decoder	min-pair mean ROUGE-L
Basic-UWM (no drafter, vacuous bound)	1.000
PFR (ours)	0.990
MWS	0.922
PFR-NoWM (no wm)	0.436
MSE-Pseudo	0.426
MSE	0.406

Appendix

Table 15: Output stability under drafter substitution. Pairwise ROUGE-L F1 between the realised token sequences across each

(D_{i},D_{j})

pair, per-prompt minimum averaged over prompts. An unbiased verify rule produces identical sequences across all drafters, so ROUGE-L

\to 1

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Oct 22, 2026

Speculative Diffusion is a Tale of Two Tradeoffs

Speculative decoding is a technique that accelerates the inference speed of autoregressive large language models (LLMs) by sampling from a faster, lower quality draft model. Traditionally, a fundamental characteristic of speculative decoding is that it is lossless with respect to the verifier distribution, guaranteeing speed without sacrificing quality. This differs from recently developed diffusion LLMs (dLLMs), which are notably able to achieve substantial speedups by degrading the quality of their outputs with less diffusion effort. Because diffusion models are able to sample faster than autoregressive models by generating tokens in parallel, they are considered strong candidates for draft models in speculative diffusion. While it is widely understood that diffusion models can sacrifice quality for speed, speculative decoding itself can also be generalized to generate lossy samples faster using more lenient acceptance criteria with respect to the verifier distribution. Presented with the two parameters of diffusion effort and speculative decoding leniency, this paper explores the full achievable Pareto frontier via joint variations of these parameters. Our experiments show that jointly varying diffusion effort and acceptance leniency can extend the speed-quality Pareto frontier over single step diffusion drafting.

Aaron Zorel, Guy Van den Broeck, Aditya Grover +1

Independent Researcher · University of California, Los Angeles

LLM Inference Speculative Decoding Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding

Ask PDF

Jul 14, 2026

WaterMoE: Expert-Routing-based Watermarking for High Fidelity and Efficiency

Large language models (LLMs) have achieved remarkable success but raise growing concerns about content provenance and misuse, motivating the need for reliable watermarking techniques. However, these techniques have rarely been adopted in practice mainly for two reasons: i) severely degraded model performance, and ii) additional inference overhead. To confirm the problem, we construct a comprehensive benchmark spanning different generation tasks to systematically evaluate 9 representative watermarking methods. We found almost all existing methods are designed for text fluency, but not for restricted and complicated tasks, and their overhead prevents them from deployment in latency-critical systems. To address i) and ii), we propose an LLM watermarking scheme WaterMoE for the growingly popular Mixture-of-Experts (MoE) LLMs. WaterMoE embeds watermarking signals through controlled perturbation into the expert selection at each router, which accumulates to token selection shift at the final output. In contrast to watermarking as a post-processing token-sampling approach, WaterMoE embeds watermark within the inference loop incurring negligible quality degradation and computational overhead. Extensive experiments demonstrate that our method achieves a fidelity performance close to the unwatermarked and consistently outperforms state-of-the-art watermarking methods on the benchmark, with up to

4\times

speedup, incurring merely 1% additional inference latency compared to native generation. The results demonstrate the capability of WaterMoE to be deployed in real-world tasks.

Z Sun, Q Jiang, S Sheng +1

Shanghai Jiao Tong University · Shanghai Innovation Institute · Tianjin University

LLM Inference Efficiency LLM Watermarking

Ask PDF

Aug 25, 2026

ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping

The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model's predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92

\times

speedup over state-of-the-art multi-candidate methods. Code is available at https://github.com/Czzzk/Resispec.

Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao +2

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China

Speculative Decoding LLM Inference Acceleration

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Speculative Diffusion is a Tale of Two Tradeoffs

WaterMoE: Expert-Routing-based Watermarking for High Fidelity and Efficiency

ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping