Decoupling Is Not Identification: Supervised Evidential Learning in Next-Token Prediction

Ge Wang

cs.AIAug 14, 2026

Decoupling Is Not Identification: Supervised Evidential Learning in Next-Token Prediction

Authors: Ge Wang

Organizations: Biomedical Imaging Center, Center for Biotechnology and Interdisciplinary Studies, Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

Abstract

A next-token probability says what a model predicts, not how much training support lies behind it. A Dirichlet head can represent this distinction by separating mean $m$ from concentration $S$ , but decoupling does not identify what $S$ means. Here we propose an Evidential Next-Token Prediction (ENTOP) framework to audit this gap on character-level Moby-Dick, using exact 8-gram count as a reproducible lexical-support label and withholding count regression from 20% of context types. Standard implicit evidential training carries essentially no count signal beyond confidence on held-out-label types (partial Spearman $ρ= 0.001 \pm 0.014$ ), whereas explicit supervision generalizes ( $ρ= 0.201 \pm 0.010$ ; matched-pair win $= 0.822 \pm 0.021$ ). CE predictive entropy is at chance for unseen 8-grams (AUROC $= 0.490 \pm 0.004$ ), while supervised vacuity reaches $0.772 \pm 0.003$ , comparable with an indexed CE-representation baseline ( $0.769$ ) but below the tautological corpus oracle ( $1.000$ ). Neither longest-suffix nor representation-distance strata explain where amortization succeeds. Increasing count weight under the digamma objective improves support fit only by sacrificing prediction. A constant predictor wins natural log-RMSE, and vacuity does not improve error deferral. These results motivate a minimum evidence protocol---confidence control, matched pairs, held-out labels, a constant baseline, and a decision test---and show that concentration can pass identification while failing calibration and utility.

Figures & tables

Panel of Figure 1 — Figure 1: ENTOP–S assigns separate targets to prediction direction and evidence strength. The held-out 20% receive language-model training but no count regression.

Model	loss	head	count	Acc.	NLL	sup- $\rho$	held- $\rho$	sup-pair	held-pair	boundary
CE	CE	–	–	$0.487\pm 0.002$	$1.687\pm 0.007$	–	–	–	–	–
CE–S	CE	exact	no	$0.487\pm 0.004$	$1.688\pm 0.007$	$0.000\pm 0.013$	$0.001\pm 0.025$	$0.506\pm 0.014$	$0.515\pm 0.033$	$0.507\pm 0.036$
ENTOP–S	CE	exact	yes	$0.483\pm 0.002$	$1.714\pm 0.007$	$0.201\pm 0.007$	$0.201\pm 0.010$	$0.888\pm 0.011$	$0.822\pm 0.021$	$0.772\pm 0.003$
ENTOP–D	Digamma+KL	exact	no	$0.445\pm 0.015$	$2.631\pm 0.053$	$0.003\pm 0.016$	$0.001\pm 0.014$	$0.545\pm 0.019$	$0.534\pm 0.021$	$0.508\pm 0.011$
D+count	Digamma+KL	exact	yes	$0.441\pm 0.007$	$2.719\pm 0.031$	$0.014\pm 0.018$	$0.022\pm 0.017$	$0.595\pm 0.025$	$0.583\pm 0.048$	$0.562\pm 0.010$
Coupled	Digamma+KL	coupled	no	$0.489\pm 0.002$	$2.554\pm 0.009$	$-0.047\pm 0.002$	$-0.036\pm 0.007$	$0.555\pm 0.009$	$0.553\pm 0.021$	$0.522\pm 0.005$

Table 1: Five-seed mean

\pm

sample SD. sup/held denote supervised/held-out types.

\rho

denotes partial Spearman correlation. Pair sets are fixed across models. Boundary is vacuity AUROC for unseen test 8-grams.

	natural		bin-balanced		$n\geq 20$
Model	model	const.	model	const.	model	const.
CE–S	$0.450\pm 0.037$	$0.304\pm 0.000$	$1.835\pm 0.112$	$1.673\pm 0.000$	$1.830\pm 0.095$	$1.611\pm 0.000$
ENTOP–S	$0.575\pm 0.031$	$0.304\pm 0.000$	$\mathbf{1.030\pm 0.048}$	$1.673\pm 0.000$	$\mathbf{0.963\pm 0.043}$	$1.611\pm 0.000$
ENTOP–D	$2.332\pm 0.037$	$0.304\pm 0.000$	$\mathbf{1.568\pm 0.038}$	$1.673\pm 0.000$	$\mathbf{1.036\pm 0.069}$	$1.611\pm 0.000$
D+count	$2.015\pm 0.022$	$0.304\pm 0.000$	$\mathbf{1.293\pm 0.021}$	$1.673\pm 0.000$	$\mathbf{0.697\pm 0.027}$	$1.611\pm 0.000$
Coupled	$2.162\pm 0.003$	$0.304\pm 0.000$	$\mathbf{1.394\pm 0.001}$	$1.673\pm 0.000$	$\mathbf{0.799\pm 0.004}$	$1.611\pm 0.000$

Table 2: Held-out log-RMSE. “Natural” weights every type equally; “balanced” weights log-count bins equally. Each model is compared with the same constant predictor; bold model values beat that constant.

Figure 2: Support identification and boundary detection. Error bars show sample SD over five seeds. The corpus oracle is exact by construction; the CE-

h

comparator stores a 50,000-type external bank.

$\beta$	Acc.	NLL	sup- $\rho$	held- $\rho$	held-pair	boundary
0.1	0.432	2.699	0.028	0.017	0.588	0.553
0.3	0.429	2.769	0.029	0.038	0.520	0.548
1	0.415	2.900	0.033	0.040	0.593	0.597
3	0.387	3.105	0.069	0.087	0.706	0.640
10	0.336	3.315	0.132	0.133	0.809	0.711

Table 3: Seed-0 diagnostic D+count sweep. It is a trade-off trace, not a five-seed model comparison.

Figure 3: Claim boundaries. (a) Natural RMSE favors a constant, while balanced and high-count evaluations favor ENTOP–S. (b) Vacuity fails selective error prediction. (c–d) Neither suffix length nor CE-

h

proximity yields a monotonic support-association gradient.

Explore similar work

NITP: Next Implicit Token Prediction for LLM Pre-training
May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model Pretraining Representation Geometry
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1
When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming
May 22, 2026Francesco CorielliAutoregressive Language Modeling Next-Token Prediction
Information-Theoretic Analysis of Next-Token Prediction under Markovian Data
Sep 28, 2026Masoud Kavian, Abdellatif Zaidi, Milad SefidgaranInformation-Theoretic Generalization Bounds Next-Token Prediction
Mathematical and Algorithmic Science Laboratory, Huawei Paris Research Center, 92100 Boulogne-Billancourt, France · Laboratoire d’Informatique Gaspard Monge, Université Gustave Eiffel, 77420 Champs-sur-Marne, France

May 24, 2026J/K move · Enter open · S save

NITP: Next Implicit Token Prediction for LLM Pre-training

Xiangdong Zhang, Debing Zhang, Shaofeng Zhang, Xiaohan Qin+2

School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China+1

Language Model Pretraining Representation Geometry Self-Supervised Pre-Training Next-Token Prediction

Open paper AskPDF arXiv

Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that can limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense continuous supervision directly in the representation space. NITP trains the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. We provide theoretical analysis showing that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and encouraging a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. On a 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with approximately 2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.

Abstract

Explore similar work

NITP: Next Implicit Token Prediction for LLM Pre-training

When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming

Information-Theoretic Analysis of Next-Token Prediction under Markovian Data