Cross-tokenizer on-policy distillation (OPD) requires aligning teacher and student sequences before their scores can be compared. Decoded-text alignment provides such correspondence at the chunk level, but leaves a distinct problem unresolved: credit alignment. A chunk-level signal specifies how much a span should change, but not how that signal should be distributed across student tokens, while tokens without teacher correspondence remain unconstrained. We propose CAST (Chunk-aware credit Assignment with Selective Token Anchoring), a shift-or-stay framework that addresses both issues. For aligned chunks, CAST redistributes each chunk's advantage across student tokens using predictive uncertainty and local teacher-signal smoothness, emphasizing learning opportunities supported by locally consistent teacher feedback while preserving the chunk's total advantage. The resulting weights exactly solve a KL-regularized credit-allocation problem. For unaligned positions, CAST selectively anchors the student to its initial policy through a KL penalty, limiting drift where teacher supervision is unavailable. Because credit assignment is separated from chunk construction, CAST also applies to punctuation-defined or fixed-length chunks under shared tokenization. Across six mathematics, science, and code benchmarks, CAST improves the average over the strongest evaluated baseline by 2.45 and 1.32 points for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and Llama-3.1-8B, respectively, and by 2.10 points for shared-tokenizer transfer to Qwen3-1.7B.
Figures & tables
Figure 1: Chunk-level alignment does not resolve credit alignment. (a) Alignment-only propagation uniformly distributes each chunk’s advantage across aligned student tokens, leaving unaligned tokens unconstrained. (b) CAST assigns credit using student uncertainty and local teacher-signal smoothness, while anchoring unaligned tokens to the supervised fine-tuning (SFT) reference with selective KL regularization.
Figure 2: Tokenizer mismatch persists despite vocabulary overlap. (a) Teacher vocabulary with a decoded-content match in the student. (b) Student response tokens without one-to-one teacher correspondence under heterogeneous decoded-text alignment (Appendix A.1 ). The shared-tokenizer 3.6% residual results from decode/re-encode normalization and special/boundary-token handling; training instead uses identity correspondence.
Figure 3: CAST: token-level shift-or-stay optimization. Shift normalizes uncertainty ui times smoothness si to allocate token credit while preserving each chunk’s advantage. Stay anchors unaligned predictions to the initial policy. Heterogeneous pairs use decoded-text alignment and fixed-length credit chunks; shared-tokenizer training uses punctuation-defined chunks here and regularizes all positions.
Table 5: Three-seed Mistral pass@1 (%, lr 10−6 ): mean ± sample SD of highest-Avg-6 evaluated checkpoints (one per seed). Per-seed results and selection: Appendix E .
Figure 4: Gradient consistency across four credit rules. Sketched Mistral LM-head gradients use 48 shared rollouts per checkpoint (32 train, 16 held out). (a) Held-out cosine with each rule’s training mean at SFT initialization (mean ± SD). (b) SNR. (c) Training–held-out mean cosine. Error bars in b,c: 95% bootstrap CIs. Protocol: Appendix B.5 .
Table 7: Shared-tokenizer teacher–student scaling results from raw Qwen3 instruction models. The method column identifies results trained with CAST.
Figure 6: CAST redistributes credit to individual tokens. (a) Uniform shares, uncertainty ui , smoothness si , and CAST weights wi for a six-token chunk from a Qwen3-8B → Mistral-7B rollout. (b) CAST-to-uniform credit-share ratios by token category across all 171 chunks of the same rollout; ratios above 1 indicate upweighting.
Figure 7: Smoothness primarily reflects teacher structure rather than projection geometry. (a) On cross-tokenizer rollouts (Mistral ← Qwen3-8B), 79% of the credit re-weighting induced by smoothness-only allocation, ∣wiS−1/Kg∣ , falls on teacher-variation positions (alignment-chunk boundaries and 1:1 positions), while 21% falls at projection-flat alignment-chunk interiors. Alignment chunks containing multiple student tokens cover 19% of tokens; the ϵ floor is active in 36% of credit chunks. (b) Cross-tokenizer projection is compared with shared-tokenizer Qwen3-8B → Qwen3-4B, where M=N and no projection is used. The shared-tokenizer setting has more credit chunks whose mean variation exceeds the floor and a larger median δ . Both panels are computed offline from saved rollouts and teacher scores, without additional training.
Hyperparameter
Value
Teacher / students
Qwen3-8B / {Mistral-7B, Llama-3.1-8B}
Student initialization
SFT on OpenThoughts 400K
Prompt-set sampling seed (v1 / v2)
44
Optimizer learning rate
1×10−6
LR warmup
10 steps
Weight decay
0.01
Appendix
Table 9: Training hyperparameters for CAST (cross-tokenizer setting).
Method
Seed
Step
A24
A25
A26
M500
GPQA
LCB
Avg.
DPCA
44
150
38.3
37.1
44.4
73.7
35.7
16.0
40.87
653512071
175
41.8
35.8
47.7
73.8
35.4
13.9
41.40
893671858
175
41.0
34.6
45.0
73.8
34.1
9.8
39.72
Uniform credit
44
150
45.0
35.2
47.1
78.0
36.5
26.5
44.72
653512071
175
42.3
37.2
46.7
78.9
38.5
26.8
45.07
893671858
200
45.8
35.9
44.8
76.6
36.5
28.3
44.65
Appendix
Table 10: Performance across three evaluated training seeds per configuration for Qwen3-8B → Mistral-7B at learning rate 10−6 . Per-seed pass@1 (%); Step identifies each seed’s highest-Avg-6 evaluated checkpoint. Aggregate means and sample standard deviations appear in Table 6 .
Method
Seed
Step
A24
A25
A26
M500
GPQA
LCB
Avg.
ULD
44
200
41.6
34.5
41.9
75.1
33.1
25.6
41.97
653512071
225
41.6
36.2
41.9
75.1
34.7
25.8
42.55
893671858
175
41.0
34.1
40.2
73.7
35.7
25.6
41.72
CAST
44
250
42.7
37.2
43.8
74.1
36.9
26.6
43.55
653512071
250
44.1
37.7
45.0
75.0
34.5
26.9
43.87
893671858
200
43.1
39.0
41.7
75.2
36.7
26.7
43.73
Appendix
Table 11: Per-seed Llama-3.1-8B pass@1 (%) with Qwen3-8B as teacher. Step identifies the reported checkpoint; summary statistics appear in Table 6 .
Method
Seed
Collapse rate
44
45
46
653512071
893671858
417698746
DPCA (vanilla)
✓
✓
✓
✓
✓
✓
0/6 ( 0.0% )
Uniform credit
✓
× (RC)
× (EC)
✓
✓
× (EC)
3/6 ( 50.0% )
Uncertainty only
✓
× (RC)
× (EC)
✓
✓
✓
2/6 ( 33.3% )
CAST
✓
✓
× (RC)
✓
✓
✓
1/6 ( 16.7% )
Appendix
Table 12: Training collapse across six seeds (Qwen3-8B → Mistral-7B, lr 10−6 ). ✓ : completed 250 steps with healthy scores; × : repetition collapse (RC) or entropy collapse (EC). Rates count collapsed runs out of six.
Figure 8: CAST has slightly broader effective credit support than uncertainty-only weighting in seed- 44 diagnostic runs. Using the seed- 44 diagnostic trajectories from Figure 5 , we compare uniform credit, uncertainty-only weighting, and CAST. For each chunk, the effective token count is the inverse participation ratio Neff(g)=1/∑i∈gwi2 , with normalized weights wi=Ai/Ag . It equals Kg under uniform allocation and approaches 1 as credit concentrates on one token, rather than counting tokens above a threshold. The monitor averages this quantity over chunks with Kg>1 and ∣Ag∣>10−8 ; faint traces show the exported training-step values, and the overlaid curves show five-step moving averages.
On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose Simple Cross-Tokenizer OPD (SimCT), which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at https://github.com/sunjie279/SimCT-.
Jie Sun, Mao Zheng, Mingyang Song +6
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.
On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher--student pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.