OrderGPT: Generation Order as a Learnable Modeling Choice

Chenhe Yang; Yuhuan Lyu; Tianyang Hu

cs.AIOct 19, 2026

OrderGPT: Generation Order as a Learnable Modeling Choice

Authors: Chenhe Yang, Yuhuan Lyu, Tianyang Hu

Organizations: SAGA Lab, The Chinese University of Hong Kong, Shenzhen

Abstract

Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling. Rather than competing methods, these are the two extreme points of a single axis -- the generation order. For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored. We introduce OrderGPT, which builds on Any-Order GPT to learn a task-level generation order without canonical-order supervision. Our key observation is that predicting variables from randomly sampled reveal prefixes produces recurring attention patterns in the backbone that provide a model-dependent signal of predictive dependencies. We aggregate this signal into a reciprocal dependency graph and use a spectral teacher to derive continuous generation priorities. A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture within a single training run. Training gradually shifts toward the learned order, which is then frozen for subsequent autoregressive training and generation. The same mechanism adapts across modalities: adaptively it recovers near-L2R dependency structure in language, learns anatomically organized factorizations on human pose, and discovers a factorization on tabular data that places statistically and semantically related fields nearby, while improving downstream modeling metrics across all domains. The code is available at https://github.com/Chenhe-yang/OrderGPT/.

Figures & tables

Figure 1: OrderGPT augments the AO-GPT backbone with an Order MLP. The Spectral Teacher constructs a reciprocal affinity graph from aggregated AO-GPT attention, recovers a Fiedler axis, and orients it using directed attention. The resulting continuous priorities supervise the Order MLP, whose outputs are sorted to obtain the task-level generation order used by the backbone.

Figure 2: Attention patterns under the current and original indexing. (a, b) WikiText103 Seq80 attention. (c, d) H3WB attention. The original-frame panels reorder the same values only for visualization, highlighting local tokens and anatomical structure. Each current/original pair shares a color scale.

Figure 3: Order MLP effectiveness on WikiText103 Seq80. (a) Validation loss under Order MLP, Spectral Teacher, Random-per-sample, Fixed-random AR, and Canonical AR. Vertical guides mark MLP distillation (10k–18k), annealing of the learned-order (18k–32k), and full use of the frozen learned order after 32k. The final learned axis has a positive post-hoc original-frame Kendall’s

\tau=0.999

. (b) Distillation MSE between Order-MLP priorities and oriented Fiedler targets.

Published L2R evaluation
Model and inference order	LAMBADA	WikiText-2	PTB	WikiText-103	1BW
SEDD	49.410	41.190	118.740	41.700	72.600
RADD	49.090	38.260	107.780	38.410	63.330
GPT-2	45.040	42.430	138.430	41.600	75.200
$\sigma$ -GPT †	68.610	57.660	146.870	55.540	90.980
AO-GPT ‡	52.460	42.100	135.960	40.970	71.730

Table 1: Zero-shot perplexity (

\downarrow

) on language benchmarks for approximately 125M-parameter models. Published baselines are reproduced from Table 5 of AO-GPT ( Xue et al., 2025 ) . † denotes a model reproduced by the AO-GPT authors, and ‡ denotes AO-GPT trained with an additional 10% of L2R-ordered data. ∗ denotes our AO-GPT baseline trained for 400k updates without language-model EMA or a prescribed L2R training mixture. Published rows follow their original settings and provide numerical context rather than a compute-matched ranking. Fixed-random AR (throughout) uses one presampled random permutation throughout training and inference. Fixed-random AR + curriculum follows OrderGPT’s random-order warm-up and order-use annealing schedule, with a presampled random permutation as its terminal order for subsequent training and inference.

Method	Training data	NLL/joint $\downarrow$	Whole $\downarrow$	Body $\downarrow$	Face-local $\downarrow$	Hand-local $\downarrow$
Large SimpleBaseline	H3WB	–	131.40	131.60	19.80	44.80
Jointformer	H3WB	–	109.20	103.00	19.80	53.50
Jointformer	H3WB+T3WB	–	84.20	80.10	16.30	44.50
Canonical AR	H3WB	$-11.05\pm 0.02$	$80.83\pm 1.80$	$74.86\pm 0.59$	$10.87\pm 0.66$	$37.41\pm 3.87$
Fixed-random AR	H3WB	$-10.68\pm 0.09$	$99.26\pm 11.87$	$82.69\pm 8.53$	$13.14\pm 2.68$	$36.36\pm 1.71$
AO-GPT	H3WB	$-10.59\pm 0.07$	$115.23\pm 15.98$	$89.97\pm 8.40$	$14.67\pm 2.95$	$38.18\pm 0.42$

Table 2: Performance on H3WB masked-I2D. Internal results are reported as mean

\pm

sample standard deviation across three independently trained model seeds (

n=3

per method). Each run is evaluated on the same 10,000-pose test split. NLL is averaged per joint, and MPJPE values are in millimeters. Published results are taken from Table 4 of Zhu et al. (2023) and provide external numerical context. Lower is better for every metric.

Published references
Method	Test NLL/variable ( $\times 10^{-3}$ ) $\downarrow$	Q-error P50 $\downarrow$	Q-error P99 $\downarrow$	Maximum Q-error $\downarrow$
Progressive Sampling	–	$1.030\pm 0.010$	$1.490\pm 0.080$	$8.860\pm 14.000$
Variable Skipping	–	$1.090\pm 0.010$	$2.480\pm 0.290$	$14.700\pm 7.000$
Multi-Order + Skipping	–	$1.200\pm 0.240$	$3.200\pm 0.290$	$12.600\pm 5.000$
Internal evaluations
Canonical AR	$325.818\pm 0.020$	$1.029\pm 0.003$	$1.459\pm 0.007$	$2.862\pm 1.137$

Table 3: Performance in density modeling and cardinality estimation on CENSUS. Internal results are reported as mean

\pm

sample standard deviation across three independently trained model seeds (

n=3

per method). Test NLL is averaged per variable on the held-out split and displayed in units of

10^{-3}

. Published rows reproduce the reported 10,000-particle results ( Yang et al., 2019 ; Liang et al., 2020 ) . Internal Q-error uses the same 1,000-query workload with 10,000 particles per query. Lower is better, and bold denotes the lowest internal mean.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Seq80	OpenWebText	H3WB	CENSUS
Ordered variables $D$	80	1,024	133	68
Decoder architecture	$4/8/384$	$12/10/640$	$4/8/256$	$2/4/128$
Global batch	512	480	160	512
Backbone updates	50,000	400,000	100,000	50,000
Peak learning rate	$10^{-3}$	$6\times 10^{-4}$	$3\times 10^{-4}$	$3\times 10^{-4}$
Minimum learning rate	$10^{-4}$	$6\times 10^{-5}$	$3\times 10^{-5}$	$3\times 10^{-5}$

Appendix

Table S1: Backbone settings: architecture denotes decoder layers / heads / hidden width, and batch counts examples per optimizer update.

Setting	Seq80	OpenWebText	H3WB	CENSUS
Examples per attention window	1,024	2,400	1,280	1,024
Updates per window	2	5	8	2
Start distillation	10k	4k	15k	5k
Stop distillation	18k	40k	40k	10k
Annealing complete	32k	60k	60k	20k

Appendix

Table S2: Attention windows and three order-learning milestones.

Figure S1: Loss trajectories: (a) OpenWebText training token NLL ; (b) H3WB validation NLL per joint; (c) CENSUS validation NLL per variable. Vertical guides mark curriculum transitions. The OpenWebText Fixed-random AR + curriculum curve follows the same schedule as OrderGPT. Internal curves represent individual runs and are averaged or smoothed for display.

Figure S2: Post-hoc interpretation of one learned H3WB order. (a) Reveal rank on a representative training-set 2D pose, with face and hand enlargements; darker points appear earlier. (b) Anatomical regions along canonical and learned orders. The pose and connecting lines provide display context, not an input-specific prediction, learned dependency edges, or the evaluation bone set. The pose, canonical identities, and anatomical labels do not enter order learning, direction selection, or model selection.

Neighborhood or check	Observed fields and positions
Service/military, 1–10	iOthrserv , iMay75880 , iKorean , iWWII , iRvetserv , dYrsserv , iSept80 , iFeb55 , iVietnam , iMilitary .
Family, 13–14, 21–22, 28–29	iRelat1 – iFertil ; iRownchld – iMarital ; iRspouse – iRrelchld .
Occupation/labor, 31–39	dOccup , dHour89 , dIndustry , iMobility , iTmpabsnt , dDepart , dHours , iYearwrk , iRemplpar .
Language/origin, 57–62	dAncstry2 , iCitizen , dAncstry1 , iSubfam1 , iEnglish , dPOB .
Pairwise distances	iRvetserv – dYrsserv : $1$ ; dOccup – dIndustry : $2$ ; dIncome1 – dRearning : $24$ .

Appendix

Table S3: Representative CENSUS neighborhoods in the audited order; positions are one-based. Column names and semantic groups are post-hoc annotations unavailable to the order learner, and the windows need not form a disjoint partition.

Autoregressive Modeling

Autoregressive modeling predicts sequences step by step from prior context, spanning language, time series, images, audio, and scientific simulations.

See all 82 papers

Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
Oct 31, 2025Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai +3Autoregressive Language Modeling Sequence Prediction
National Taiwan University
From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Jul 29, 2026Seunggeun Kim, Jaeyeon Kim, Taekyun Lee +4Masked Diffusion Models Diffusion Language Model Inference
University of Texas at Austin · Harvard University
Variational Learning for Insertion-based Generation
Jun 1, 2026Yangtian Zhang, Zhe Wang, Arthur Gretton +4Non-Autoregressive Text Generation
1Yale University · 2Google DeepMind · University College London +1

Oct 31, 2025J/K move · Enter open · S save

Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap

Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai, Yan-Yu Chen+2

National Taiwan University

Autoregressive Language Modeling Sequence Prediction Autoregressive Transformer

Open paper AskPDF arXiv

Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S;t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S;t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.

Abstract

Autoregressive Modeling

Explore similar work

Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap

From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

Variational Learning for Insertion-based Generation