Trajectory-Triggered Sparse Collaborative Decoding

Amit Ranjan Trivedi; Sruthikesh Surineni; Kyong Ho Lee; Mohit Mittal; Steven Li; Rex Hill

cs.CLOct 16, 2026

Trajectory-Triggered Sparse Collaborative Decoding

Authors: Amit Ranjan Trivedi, Sruthikesh Surineni, Kyong Ho Lee, Mohit Mittal, Steven Li, Rex Hill

Organizations: University of Illinois Chicago · Meta

Abstract

Dense assistance methods such as speculative decoding repeatedly verify draft tokens with a target model, keeping target computation coupled to generation. We study sparse collaborative decoding, where a draft model generates most tokens and a target model is invoked only for short corrective segments. The key problem is deciding when to call the target. We show that local uncertainty is a weak trigger: entropy, negative log-likelihood, logit margin, top- $k$ mass, and velocity features give near-chance failure prediction, with AUC 0.50-0.53. In contrast, aggregate trajectory-shape features over the draft confidence path reach AUC 0.86 using a lightweight logistic predictor trained only from draft correctness labels. This induces a trigger-timing tradeoff: early calls have more leverage but weaker evidence, while later calls have stronger evidence but less ability to redirect the trajectory. Used online, learned sparse collaborative decoding improves over the best open-loop schedule by +5.3 to +20.0 accuracy points across five GSM8K draft-target pairs. On MATH-500, it gains 7.0-13.5 points when early prediction and target competence align, but loses when either is weak. On StrategyQA, it improves over open-loop by 5.0-8.3 points, showing transfer beyond arithmetics.

Figures & tables

Figure 1: Trajectory-triggered sparse collaborative decoding. A draft model generates most of the reasoning trajectory, while a stronger target model is invoked only at selected points. Triggers use the evolving confidence trajectory rather than fixed schedules or local uncertainty spikes. Early calls have higher leverage but weaker evidence; later calls have less leverage.

Signal or feature set	AUC	Type
Entropy $H_{t}$	0.531	local
NLL $\mathrm{NLL}_{t}$	0.528	local
Margin $M_{t}$	0.474	local
Entropy velocity	0.503	local change
NLL velocity	0.499	local change
Margin velocity	0.498	local change

Table 1: Failure prediction on GSM8K with Qwen-1.5B. Local token-level signals are weak, while aggregate trajectory-shape features are strongly predictive.

Figure 2: Time-frequency view of trajectory instability. Wavelet transforms of logit-margin-derived signals show elevated mid-frequency oscillations for incorrect trajectories, consistent with unresolved competition between reasoning paths before commitment. Correct trajectories show smoother consolidation.

Draft	Dataset	Fail%	@16	@128	Full
Qwen 1.5B	GSM8K	26	0.60	0.82	0.86
Llama 1B	GSM8K	52	0.61	0.76	0.80
Qwen 0.5B	GSM8K	52	0.63	0.69	0.78
Qwen 0.5B	MATH-500	76	0.45	0.57	0.75
Llama 1B	MATH-500	86	0.56	0.59	0.73
Qwen 1.5B	MATH-500	58	0.50	0.58	0.78

Table 2: Early predictor reliability across draft and dataset pairs. AUC@128 is the operational quantity for online triggering; full AUC is more relevant to posthoc or cascade policies.

Figure 3: Intervention leverage decreases with decoding position. We perturb the draft prefix at different token positions and measure whether the final decoded sequence changes. Single-token and eight-token perturbations are most influential early and decay with position. Eight-token perturbations raise the absolute flip rate but preserve the same trend. This supports the trigger-timing tradeoff: early calls have greater influence but weaker evidence, while later calls have stronger evidence but less ability to redirect.

Draft	$K$	Mean acc	Tgt tok	Best cell
Qwen 1.5B	8	75.8	8.3	77.2
Qwen 1.5B	32	78.6	33.2	81.5
Qwen 1.5B	64	80.5	65.9	83.5
Qwen 1.5B	128	84.9	126.2	86.5
Llama 1B	8	48.8	8.4	50.0
Llama 1B	32	55.4	30.9	58.8

Table 3: Open-loop SCD on GSM8K. Per-

K

marginals average over schedule positions

P

. Best cell reports the best position for each prefix length.

Draft	$J$	Best acc	Rescue	Break	Tgt tok
Qwen 1.5B	8	80.8	10.8	4.5	33.9
Qwen 1.5B	32	87.5	15.2	2.2	97.6
Qwen 1.5B	64	91.0	17.5	1.0	122.3
Qwen 1.5B	96	91.8	18.2	1.0	141.3
Llama 1B	8	50.5	11.8	9.0	49.4
Llama 1B	32	77.8	33.2	3.2	151.7

Table 4: Closed-loop J-sweeps on GSM8K. Each row reports the best accuracy at that segment length over tested thresholds.

Panel of Figure 4 — Figure 4: Closed-loop intervention recovers trajectory alignment. We plot cumulative BLEU ( Papineni et al., 2002 ) between the draft trajectory and a target-guided reference. Draft-only decoding drifts and does not recover. Triggered target insertions occur after alignment degradation and induce rapid recovery, showing that sparse calls can redirect the trajectory.

Draft	$\theta_{e}$	$\theta_{l}$	Acc	Rescue	Tgt tok
Qwen 1.5B	0.5	0.3	92.2	19.0	151.6
Qwen 1.5B	0.5	0.5	90.8	18.0	135.1
Qwen 1.5B	0.7	0.7	88.0	14.8	90.7
Llama 1B	0.3	0.5	90.0	44.0	232.0
Llama 1B	0.5	0.5	86.0	41.2	195.7
Llama 1B	0.7	0.7	77.2	32.0	136.9

Table 5: Position-dependent threshold sweeps on GSM8K with

J=96

\theta_{e}

and

\theta_{l}

are early- and late-region thresholds. Lower thresholds fire more often, increasing rescue and target use.

Draft	Target	D-only	Best OL	Best CL
GSM8K
Qwen 1.5B	4o-mini	74.5	86.5 (104)	91.8 (141)
Llama 1B	4o-mini	47.8	75.0 (91)	88.8 (195)
Qwen 0.5B	4o-mini	48.0	69.5 (118)	89.5 (224)
Qwen 0.5B	4o	48.0	72.5 (116)	91.0 (196)
Llama 1B	4o	45.0	68.5 (66)	86.5 (192)

Table 6: Generalization across math and stress-test triples. Parentheses report average target-generated tokens per example.

Draft	D-only	Best OL	Best CL	$\Delta$
Qwen 0.5B	50.0	50.0	58.3	+8.3
Llama 1B	48.3	60.7	68.7	+8.0
Qwen 1.5B	52.3	62.3	67.3	+5.0

Table 7: Generalization to commonsense multi-hop QA on StrategyQA (

n=300

). All rows use gpt-4o-mini.

Method	Acc	Tgt tok	Calls	Wall s
Draft-only	47.8	0.0	0.00	9.0
Target-only	93.8	274.5	1.00	6.8
Self-consistency $k=5$	46.5	0.0	0.00	49.4
Posthoc rewrite	59.8	147.6	0.61	10.0
Open-loop SCD	75.0	90.6	0.90	6.3
Cascade	87.8	184.8	0.61	10.5

Table 8: Seven-method comparison on Llama-1B with gpt-4o-mini on GSM8K. Closed-loop SCD gives the best accuracy below target-only.

Draft	Dataset	auc @128	Full auc	$\Delta_{\text{CL-OL}}$
Qwen 1.5B	GSM8K	0.82	0.86	$+5.3$
Llama 1B	GSM8K	0.76	0.80	$+13.8$
Qwen 0.5B	GSM8K	0.69	0.78	$+20.0$
Qwen 0.5B	MATH-500	0.57	0.75	$-5.5$ (4o-mini)
Qwen 0.5B	MATH-500	0.57	0.75	$+7.0$ (4o)
Qwen 1.5B	MATH-500	0.58	0.78	$-4.5$

Table 9: Predictor-side failure mode. Triples where the early trigger is unreliable lose under closed-loop SCD, while triples with

\textsc{auc}@128\geq 0.70

win. Pairs identical drafts on different targets isolate the predictor effect (rows 3-4).

Metric	Recovery	Degradation
Mean NLL $\downarrow$	0.162	0.200
Mean entropy $\downarrow$	0.336	0.413
Mean max prob $\uparrow$	0.877	0.854
Mean margin $\uparrow$	0.795	0.765
Hedging terms	4.1	38.2
Incomplete computation	0.8	22.0

Table 10: Target behavior at intervention time. Degrading interventions are less confident and more uncertain than recovery interventions.

Data	Target	T-only	CL	Deg.	Tok
GSM8K	4o-mini	94.5	87.0	1.5	142.7
GSM8K	4.1-mini	97.5	84.0	4.5	240.9
GSM8K	4o	95.0	91.0	2.0	122.9
GSM8K	4.1	96.5	79.5	7.0	346.2
MATH	4o-mini	69.5	49.0	5.0	405.0
MATH	4.1-mini	72.0	37.5	12.0	493.4

Table 11: Target capability is not enough. Same draft, predictor, and closed-loop parameters; only the target changes. Tok is mean target tokens per intervention.

Method	GSM8K		MATH
	1.5B	0.5B	1.5B	0.5B
Draft	73.5 / 0	48.0 / 0	50.0 / 0	29.5 / 0
OL-SCD	83.5 / 87	69.5 / 109	57.5 / 95	38.0 / 125
CL-SCD low	87.5 / 120	88.5 / 409	55.5 / 198	27.0 / 336
CL-SCD best	92.0 / 330	91.0 / 501	–	–
SD	93–95 / 244	94–95 / 244	70.5–74 / 556	70.5–71.5 / 558

Table 12: SD vs. SCD on SD-feasible Qwen pairs. Cells report accuracy / target tokens. Target is Qwen-14B.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Knob	Default	Main overrides
Decoding
Open-loop prefix $K$	$32$	open-loop sweep
Segment length $J$	$16$	$J\in\{8,32,64,96\}$
Max new tokens	$1280$	StrategyQA: $256$
Sampling	greedy	self-consistency : $T{=}0.7$ , top- $p{=}0.95$ , $k{=}5$
Trigger and budget

Appendix

Table 13: Default hyperparameters for sparse collaborative decoding.

Component	Features
Base series	$\{H,\mathrm{NLL},M,vH,v\mathrm{NLL},vM,$ $\ \|vH\|,\|v\mathrm{NLL}\|,\|vM\|,p_{1},\log M,\mathrm{top5}\}$
Aggregates	$\{\max,\min,\mathrm{mean},p10,$ $\ p90,\mathrm{std},\mathrm{slope}\}$
Scalar	trajectory length
Total	$12\times 7+1=85$

Appendix

Table 14: Trajectory-shape features used by the failure predictor. For each base series, we compute the same seven prefix-level aggregates; adding trajectory length gives

12\times 7+1=85

features.

Figure 5: Predictor reliability improves with trajectory length. GSM8K predictors cross the operational $\textsc{auc}=0.70$ gate earlier, while MATH-500 predictors remain weaker at comparable trigger positions. This explains why the same closed-loop mechanism is more reliable on GSM8K and more target-dependent on MATH-500.

Draft	Data	@16	@32	@64	@96	@128	@256	Full
Qwen 1.5B	GSM8K	0.60	0.63	0.71	0.79	0.82	0.84	0.86
Llama 1B	GSM8K	0.61	0.65	0.67	0.71	0.76	0.79	0.80
Qwen 0.5B	GSM8K	0.63	0.65	0.66	0.68	0.69	0.74	0.78
Qwen 0.5B	MATH	0.45	0.47	0.53	0.55	0.57	0.64	0.75
Llama 1B	MATH	0.56	0.57	0.58	0.59	0.59	0.66	0.73
Qwen 1.5B	MATH	0.50	0.52	0.56	0.57	0.58	0.68	0.78

Appendix

Table 15: Failure-prediction AUC at different prefix lengths. Bold marks the operational online-triggering point used in the main analysis.

Figure 6: Closed-loop accuracy-cost frontier. Increasing segment length $J$ improves closed-loop SCD until target use saturates. The learned trigger improves over the best open-loop points, especially for the weaker Llama-1B draft where more errors remain recoverable.

Figure 7: Open-loop SCD depends on insertion position and segment length. Fixed schedules can improve accuracy, but the best insertion point depends on the draft and schedule. Learned triggering avoids committing to one global schedule across examples and instead adapts to trajectory-specific drift.

Draft	$J$	Acc	Rsc	Brk	Tok	Cls
Qwen 1.5B	8	80.8	10.8	4.5	33.9	1.21
Qwen 1.5B	32	87.5	15.2	2.2	97.6	1.78
Qwen 1.5B	64	91.0	17.5	1.0	122.3	2.06
Qwen 1.5B	96	91.8	18.2	1.0	141.3	2.18
Qwen 1.5B	128	91.5	18.0	1.0	153.8	2.22
Llama 1B	8	50.5	11.8	9.0	49.4	2.55

Appendix

Table 16: Full closed-loop

J

-sweep on GSM8K. Rsc = rescue rate, Brk = break rate, Tok = target tokens per example, Cls = target calls per example.

Draft / $\theta_{e}$	$\theta_{l}$
	0.2	0.3	0.5	0.7	0.8
Qwen-1.5B
0.3	91.2	91.5	91.0	90.0	88.5
0.5	91.8	92.2	90.8	88.8	87.2
0.7	90.5	90.8	89.5	88.0	86.0
Llama-1B

Appendix

Table 17: Position-dependent threshold sweep at

J=96

on GSM8K. The best setting is mildly position-dependent for Qwen-1.5B and more strongly threshold-sensitive for Llama-1B.

Figure 8: Accuracy versus average target interventions. Results are shown for SVAMP and GSM8K with Qwen-0.5B and Qwen-1.5B drafts and a fixed Qwen-14B target. Curves correspond to different target-assisted prefix (TAP) lengths, where TAP= $K$ injects the first $K$ target tokens before draft decoding resumes. Accuracy improves with a small number of interventions, then saturates or declines when intervention becomes too aggressive.

Condition	Early	Mid	Late
	0–20	20–50	50–100
Target	2.446	2.568	2.650
Draft	2.791	3.028	3.178
Draft + K48	3.079	3.176	3.192
Draft + K96	3.206	3.172	3.075

Appendix

Table 18: Attention entropy over decoding phases. Lower entropy indicates stronger attention concentration.

Condition	Entropy slope
Target	$+0.00254$
Draft	$+0.00484$
Draft + K48	$+0.00141$
Draft + K96	$\mathbf{-0.00164}$

Appendix

Table 19: Linear attention entropy slope per token. Negative slope indicates increasing attention concentration over time.

Condition	Early ratio	Late ratio	Decay
Draft + K48	0.855	0.635	$-25.8\%$
Draft + K96	0.863	0.651	$-24.6\%$

Appendix

Table 20: Prefix attention dominance and decay over decoding.

Data	Draft	Target	draft-only	best- open-loop	best- closed-loop	$\Delta_{\mathrm{CL-OL}}$
GSM8K	Qwen 0.5B	4o-mini	48.0	69.5	89.5	$+20.0$
GSM8K	Qwen 0.5B	4o	48.0	72.5	91.0	$+18.5$
GSM8K	Llama 1B	4o	45.0	68.5	86.5	$+18.0$
GSM8K	Qwen 1.5B	4o-mini	74.5	86.5	91.8	$+5.3$
GSM8K	Llama 1B	4o-mini	47.8	75.0	88.8	$+13.8$
MATH	Qwen 0.5B	4o-mini	24.5	37.5	32.0	$-5.5$

Appendix

Table 21: Full generalization grid.

\Delta_{\mathrm{CL-OL}}

is best closed-loop minus best open-loop accuracy in percentage points.

Figure 9: Target intervention behavior matters beyond target-only accuracy. Shorter target continuations are associated with better closed-loop accuracy. Stronger target-only models can degrade closed-loop performance when their continuations are long or overwrite the draft context.

Data	Target	draft-only	target-only	closed-loop	Rsc	Deg	Cap
GSM8K	4o-mini	73.5	94.5	87.0	15.0	1.5	0.64
GSM8K	4.1-mini	73.5	97.5	84.0	15.0	4.5	0.44
GSM8K	4o	73.5	95.0	91.0	19.5	2.0	0.81
GSM8K	4.1	73.5	96.5	79.5	13.0	7.0	0.26
MATH	4o-mini	42.5	69.5	49.0	11.5	5.0	0.24
MATH	4.1-mini	42.5	72.0	37.5	7.0	12.0	$-0.17$

Appendix

Table 22: Target ladder with rescue, degradation, and captured fraction. Cap is the fraction of the draft-to-target gap captured by closed-loop SCD.

Metric	Recovery	Degradation	Direction
Per-token averages during injection
Mean NLL $\downarrow$	0.162	0.200	higher is worse
Mean entropy $\downarrow$	0.336	0.413	higher is worse
Mean max prob $\uparrow$	0.877	0.854	lower is worse
Mean margin $\uparrow$	0.795	0.765	lower is worse
Uncertainty flags: fraction of interventions with at least one flagged token

Appendix

Table 23: Target-side confidence as an intervention quality signal. Metrics are computed over the injected target segment. Recovery cases correct an unstable or incorrect draft trajectory; degradation cases worsen an otherwise correct or recoverable trajectory.

Panel of Figure 10 — Figure 10: Draft tuning for target followability. Left: layer-wise sensitivity on GSM8K, where gains concentrate in early-to-mid layers, suggesting stabilization near the target-to-draft handoff. Right: layer-wise sensitivity on SVAMP, where gains peak in middle layers, suggesting improved longer-horizon retention of injected trajectories.

Config	Acc	Lat. (ms)
No adapter	0.6200	9497.3
early r8 L1+ent	0.6650	10388.1
early r8 L1	0.6575	10022.3
early r4 L1+ent	0.6425	9637.5
early r4 L1	0.6400	9467.8
mid r8 L1+ent	0.6450	9992.0

Appendix

Table 24: Followability sweep on GSM8K. Draft: Qwen2.5-1.5B-Instruct. Target: Qwen2.5-14B-Instruct. Training uses

K_{\mathrm{target}}=80

; evaluation injects

K_{\mathrm{draft}}=32

Figure 11: Speculative decoding and SCD occupy different regimes. In SD-feasible open-weight settings, speculative decoding provides the dense parity-preserving point. SCD provides sparse text-correction operating points and applies to black-box/API targets where target logits are unavailable.

Method	Acc	Tok
Qwen-1.5B / GSM8K
draft-only	73.5	0
open-loop	83.5	87
closed-loop low	87.5	120
closed-loop best	92.0	330
spec-decode	93.0–95.0	244

Appendix

Table 25: Extended SD-feasible comparison on Qwen-family pairs. Tok is mean target tokens per example.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

May 24, 2026

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.

Jinze Li, Yixing Xu, Guanchen Li +7

Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas

Speculative Decoding LLM Inference Acceleration

Ask PDF

Mar 27, 2026

TAPS: Task Aware Proposal Distributions for Speculative Sampling

Speculative decoding accelerates autoregressive generation by letting a lightweight draft model propose future tokens that a larger target model then verifies in parallel. In practice, however, draft models are usually trained on broad generic corpora, which leaves it unclear how much speculative decoding quality depends on the draft training distribution. We study this question with lightweight HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed-data variants, evaluated on MT-Bench, GSM8K, MATH-500, and SVAMP. Measured by acceptance length, task-specific training yields clear specialization: MathInstruct-trained drafts are strongest on reasoning benchmarks, while ShareGPT-trained drafts are strongest on MT-Bench. Mixed-data training improves robustness, but larger mixtures do not dominate across decoding temperatures. We also study how to combine specialized drafters at inference time. Naive checkpoint averaging performs poorly, whereas confidence-based routing improves over single-domain drafts and merged-tree verification yields the highest acceptance length overall for both backbones. Finally, confidence is a more useful routing signal than entropy: rejected tokens tend to have higher entropy, but confidence produces much clearer benchmark-level routing decisions. These results show that speculative decoding quality depends not only on draft architecture, but also on the match between draft training data and downstream workload, and that specialized drafters are better combined at inference time than in weight space.

Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna +2

King Abdullah University of Science and Technology (KAUST) · American University of Beirut (AUB)

Speculative Decoding LLM Inference Acceleration

Ask PDF

Oct 21, 2026

Understanding Target Probability Reuse in Speculative Decoding

Speculative decoding uses a target model to verify a tree of draft tokens in parallel. We study whether the resulting next-token distributions can help select the next tree. For unrestricted target models, we show that guaranteed exact reuse requires a distribution computed at the next round's complete context. We characterize when retaining this distribution requires sacrificing accepted tokens and establish conditions under which this loss persists across decoding rounds. Across six targets, we find that reranking first-token branches with a separately computed distribution at the current context improves expected acceptance over draft ranking, especially with small tree budgets. Drawing the first token from this distribution before allocating the tree further improves expected acceptance over pruning a preselected tree, even at equal mean tree size. In our experiments, both selection rules also decode faster than draft ranking when the separate target pass overlaps standalone drafting. Our findings show that making target probabilities available before tree allocation can improve the efficiency of speculative decoding. Our code is available at https://github.com/js-lee-AI/TargetRows.

Jungseob Lee, Giyoon Jang, Chailin Nam +6

Korea University · Soongsil University

Speculative Decoding Parallel Speculative Decoding Efficient Inference

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

TAPS: Task Aware Proposal Distributions for Speculative Sampling

Understanding Target Probability Reuse in Speculative Decoding