CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

Jingru Yang; Jinhang Wu; Minjing Shi; Shengfeng He; Xuyan Li; Man Tan; Biye Li

cs.LGOct 9, 2026

CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

Authors: Jingru Yang, Jinhang Wu, Minjing Shi, Shengfeng He, Xuyan Li, Man Tan, Biye Li

Organizations: Taobao & Tmall Group of Alibaba · Singapore Management University

Abstract

Cross-tokenizer on-policy distillation (OPD) requires aligning teacher and student sequences before their scores can be compared. Decoded-text alignment provides such correspondence at the chunk level, but leaves a distinct problem unresolved: credit alignment. A chunk-level signal specifies how much a span should change, but not how that signal should be distributed across student tokens, while tokens without teacher correspondence remain unconstrained. We propose CAST (Chunk-aware credit Assignment with Selective Token Anchoring), a shift-or-stay framework that addresses both issues. For aligned chunks, CAST redistributes each chunk's advantage across student tokens using predictive uncertainty and local teacher-signal smoothness, emphasizing learning opportunities supported by locally consistent teacher feedback while preserving the chunk's total advantage. The resulting weights exactly solve a KL-regularized credit-allocation problem. For unaligned positions, CAST selectively anchors the student to its initial policy through a KL penalty, limiting drift where teacher supervision is unavailable. Because credit assignment is separated from chunk construction, CAST also applies to punctuation-defined or fixed-length chunks under shared tokenization. Across six mathematics, science, and code benchmarks, CAST improves the average over the strongest evaluated baseline by $2.45$ and $1.32$ points for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and Llama-3.1-8B, respectively, and by $2.10$ points for shared-tokenizer transfer to Qwen3-1.7B.

Figures & tables

Figure 1: Chunk-level alignment does not resolve credit alignment. (a) Alignment-only propagation uniformly distributes each chunk’s advantage across aligned student tokens, leaving unaligned tokens unconstrained. (b) CAST assigns credit using student uncertainty and local teacher-signal smoothness, while anchoring unaligned tokens to the supervised fine-tuning (SFT) reference with selective KL regularization.

Figure 2: Tokenizer mismatch persists despite vocabulary overlap. (a) Teacher vocabulary with a decoded-content match in the student. (b) Student response tokens without one-to-one teacher correspondence under heterogeneous decoded-text alignment (Appendix A.1 ). The shared-tokenizer

3.6\%

residual results from decode/re-encode normalization and special/boundary-token handling; training instead uses identity correspondence.

Figure 3: CAST: token-level shift-or-stay optimization. Shift normalizes uncertainty

u_{i}

times smoothness

s_{i}

to allocate token credit while preserving each chunk’s advantage. Stay anchors unaligned predictions to the initial policy. Heterogeneous pairs use decoded-text alignment and fixed-length credit chunks; shared-tokenizer training uses punctuation-defined chunks here and regularizes all positions.

Setting	A24	A25	A26	M500	GPQA	LCB	Avg.
Uniform credit	46.6	36.7	46.5	76.5	38.3	33.9	46.42
Fixed chunks ( $K{=}6$ )	48.9	36.6	46.5	76.3	39.3	35.1	47.12
Shuffled credit	48.2	36.6	42.9	76.2	36.0	33.0	45.48
CAST	51.6	37.0	51.7	77.2	40.0	34.6	48.68

Table 3: Shared-tokenizer CAST ablations (Qwen3-8B

\rightarrow

raw Qwen3-1.7B, v1). Uniform uses equal weights; shuffled permutes CAST weights within chunks; fixed

K{=}6

replaces punctuation-defined chunks.

Method	A24	A25	A26	M500	GPQA	LCB	Avg.
DPCA	$40.4\pm 1.8$	$35.8\pm 1.3$	$45.7\pm 1.8$	$73.8\pm 0.1$	$35.1\pm 0.9$	$13.2\pm 3.2$	$40.66\pm 0.86$
Uniform credit	$44.4\pm 1.8$	$36.1\pm 1.0$	$46.2\pm 1.2$	$77.8\pm 1.2$	$37.2\pm 1.2$	$27.2\pm 1.0$	$44.81\pm 0.22$
Uncertainty only	$49.0\pm 1.5$	$40.7\pm 1.0$	$49.7\pm 1.3$	$74.7\pm 1.1$	$39.3\pm 0.8$	$17.8\pm 0.8$	$45.21\pm 0.64$
CAST	$45.6\pm 0.7$	$39.9\pm 0.9$	$50.1\pm 1.2$	$79.0\pm 1.2$	$39.5\pm 0.7$	$27.9\pm 0.2$	$47.01\pm 0.28$

Table 5: Three-seed Mistral pass@1 (%, lr

10^{-6}

): mean

\pm

sample SD of highest-Avg-6 evaluated checkpoints (one per seed). Per-seed results and selection: Appendix E .

Figure 4: Gradient consistency across four credit rules. Sketched Mistral LM-head gradients use 48 shared rollouts per checkpoint (32 train, 16 held out). (a) Held-out cosine with each rule’s training mean at SFT initialization (mean

\pm

SD). (b) SNR. (c) Training–held-out mean cosine. Error bars in b,c:

95\%

bootstrap CIs. Protocol: Appendix B.5 .

Figure 5: Mistral training dynamics. (a) Positive-teacher-advantage fraction; (b) policy entropy; (c) gradient norm; (d)

4

-gram repetition. Curves: five-step moving averages. Details: Appendix B.4 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Student	Teacher	Method	A24	A25	A26	M500	GPQA	LCB	Avg.
Qwen3-1.7B	None (raw)	–	46.2	34.0	42.5	75.9	37.8	34.1	45.08
	Qwen3-8B	CAST	51.6	37.0	51.7	77.2	40.0	34.6	48.68
	Qwen3-30B-A3B	CAST	50.2	39.4	49.0	76.5	37.9	35.3	48.05
Qwen3-4B	None (raw)	–	66.5	58.6	58.8	81.7	54.5	54.4	62.42
	Qwen3-8B	CAST	66.4	59.3	60.6	81.2	54.3	52.2	62.33
	Qwen3-30B-A3B	CAST	69.2	63.6	66.9	81.5	53.7	52.6	64.58

Appendix

Table 7: Shared-tokenizer teacher–student scaling results from raw Qwen3 instruction models. The method column identifies results trained with CAST.

Figure 6: CAST redistributes credit to individual tokens. (a) Uniform shares, uncertainty

u_{i}

, smoothness

s_{i}

, and CAST weights

w_{i}

for a six-token chunk from a Qwen3-8B

\rightarrow

Mistral-7B rollout. (b) CAST-to-uniform credit-share ratios by token category across all

171

chunks of the same rollout; ratios above

1

indicate upweighting.

Figure 7: Smoothness primarily reflects teacher structure rather than projection geometry. (a) On cross-tokenizer rollouts (Mistral

\leftarrow

Qwen3-8B),

79\%

of the credit re-weighting induced by smoothness-only allocation,

|w_{i}^{S}-1/K_{g}|

, falls on teacher-variation positions (alignment-chunk boundaries and

1{:}1

positions), while

21\%

falls at projection-flat alignment-chunk interiors. Alignment chunks containing multiple student tokens cover

19\%

of tokens; the

\epsilon

floor is active in

36\%

of credit chunks. (b) Cross-tokenizer projection is compared with shared-tokenizer Qwen3-8B

\rightarrow

Qwen3-4B, where

M=N

and no projection is used. The shared-tokenizer setting has more credit chunks whose mean variation exceeds the floor and a larger median

\delta

. Both panels are computed offline from saved rollouts and teacher scores, without additional training.

Hyperparameter	Value
Teacher / students	Qwen3-8B / {Mistral-7B, Llama-3.1-8B}
Student initialization	SFT on OpenThoughts 400K
Prompt-set sampling seed (v1 / v2)	44
Optimizer learning rate	$1\times 10^{-6}$
LR warmup	10 steps
Weight decay	0.01

Appendix

Table 9: Training hyperparameters for CAST (cross-tokenizer setting).

Method	Seed	Step	A24	A25	A26	M500	GPQA	LCB	Avg.
DPCA	44	150	38.3	37.1	44.4	73.7	35.7	16.0	40.87
	653512071	175	41.8	35.8	47.7	73.8	35.4	13.9	41.40
	893671858	175	41.0	34.6	45.0	73.8	34.1	9.8	39.72
Uniform credit	44	150	45.0	35.2	47.1	78.0	36.5	26.5	44.72
	653512071	175	42.3	37.2	46.7	78.9	38.5	26.8	45.07
	893671858	200	45.8	35.9	44.8	76.6	36.5	28.3	44.65

Appendix

Table 10: Performance across three evaluated training seeds per configuration for Qwen3-8B

\rightarrow

Mistral-7B at learning rate

10^{-6}

. Per-seed pass@1 (%); Step identifies each seed’s highest-Avg-6 evaluated checkpoint. Aggregate means and sample standard deviations appear in Table 6 .

Method	Seed	Step	A24	A25	A26	M500	GPQA	LCB	Avg.
ULD	44	200	41.6	34.5	41.9	75.1	33.1	25.6	41.97
	653512071	225	41.6	36.2	41.9	75.1	34.7	25.8	42.55
	893671858	175	41.0	34.1	40.2	73.7	35.7	25.6	41.72
CAST	44	250	42.7	37.2	43.8	74.1	36.9	26.6	43.55
	653512071	250	44.1	37.7	45.0	75.0	34.5	26.9	43.87
	893671858	200	43.1	39.0	41.7	75.2	36.7	26.7	43.73

Appendix

Table 11: Per-seed Llama-3.1-8B pass@1 (%) with Qwen3-8B as teacher. Step identifies the reported checkpoint; summary statistics appear in Table 6 .

Method	Seed						Collapse rate
Method	44	45	46	653512071	893671858	417698746	Collapse rate
DPCA (vanilla)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$0/6$ ( $0.0\%$ )
Uniform credit	$\checkmark$	$\times$ (RC)	$\times$ (EC)	$\checkmark$	$\checkmark$	$\times$ (EC)	$3/6$ ( $50.0\%$ )
Uncertainty only	$\checkmark$	$\times$ (RC)	$\times$ (EC)	$\checkmark$	$\checkmark$	$\checkmark$	$2/6$ ( $33.3\%$ )
CAST	$\checkmark$	$\checkmark$	$\times$ (RC)	$\checkmark$	$\checkmark$	$\checkmark$	$1/6$ ( $16.7\%$ )

Appendix

Table 12: Training collapse across six seeds (Qwen3-8B

\rightarrow

Mistral-7B, lr

10^{-6}

\checkmark

: completed

250

steps with healthy scores;

\times

: repetition collapse (RC) or entropy collapse (EC). Rates count collapsed runs out of six.

Figure 8: CAST has slightly broader effective credit support than uncertainty-only weighting in seed-

44

diagnostic runs. Using the seed-

44

diagnostic trajectories from Figure 5 , we compare uniform credit, uncertainty-only weighting, and CAST. For each chunk, the effective token count is the inverse participation ratio

N_{\mathrm{eff}}(g)=1/\sum_{i\in g}w_{i}^{2}

, with normalized weights

w_{i}=\widehat{A}_{i}/A_{g}

. It equals

K_{g}

under uniform allocation and approaches

1

as credit concentrates on one token, rather than counting tokens above a threshold. The monitor averages this quantity over chunks with

K_{g}>1

and

|A_{g}|>10^{-8}

; faint traces show the exported training-step values, and the overlaid curves show five-step moving averages.

Cross-Tokenizer Knowledge Distillation

Cross-tokenizer knowledge distillation transfers a language model’s behavior to a student with a different vocabulary and tokenization scheme.

See all 20 papers

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Oct 6, 2026Bingxi Hou, Guochao Jiang, Guofeng Quan +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Alibaba Cloud Computing

May 8, 2026J/K move · Enter open · S save

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong+5

University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent+1

Cross-Tokenizer Knowledge Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose Simple Cross-Tokenizer OPD (SimCT), which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at https://github.com/sunjie279/SimCT-.

Abstract

Cross-Tokenizer Knowledge Distillation

Explore similar work

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability