Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling. Rather than competing methods, these are the two extreme points of a single axis -- the generation order. For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored. We introduce OrderGPT, which builds on Any-Order GPT to learn a task-level generation order without canonical-order supervision. Our key observation is that predicting variables from randomly sampled reveal prefixes produces recurring attention patterns in the backbone that provide a model-dependent signal of predictive dependencies. We aggregate this signal into a reciprocal dependency graph and use a spectral teacher to derive continuous generation priorities. A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture within a single training run. Training gradually shifts toward the learned order, which is then frozen for subsequent autoregressive training and generation. The same mechanism adapts across modalities: adaptively it recovers near-L2R dependency structure in language, learns anatomically organized factorizations on human pose, and discovers a factorization on tabular data that places statistically and semantically related fields nearby, while improving downstream modeling metrics across all domains. The code is available at https://github.com/Chenhe-yang/OrderGPT/.
Figures & tables
Figure 1: OrderGPT augments the AO-GPT backbone with an Order MLP. The Spectral Teacher constructs a reciprocal affinity graph from aggregated AO-GPT attention, recovers a Fiedler axis, and orients it using directed attention. The resulting continuous priorities supervise the Order MLP, whose outputs are sorted to obtain the task-level generation order used by the backbone.
Figure 2: Attention patterns under the current and original indexing. (a, b) WikiText103 Seq80 attention. (c, d) H3WB attention. The original-frame panels reorder the same values only for visualization, highlighting local tokens and anatomical structure. Each current/original pair shares a color scale.
Figure 3: Order MLP effectiveness on WikiText103 Seq80. (a) Validation loss under Order MLP, Spectral Teacher, Random-per-sample, Fixed-random AR, and Canonical AR. Vertical guides mark MLP distillation (10k–18k), annealing of the learned-order (18k–32k), and full use of the frozen learned order after 32k. The final learned axis has a positive post-hoc original-frame Kendall’s τ=0.999 . (b) Distillation MSE between Order-MLP priorities and oriented Fiedler targets.
Model and inference order
LAMBADA
WikiText-2
PTB
WikiText-103
1BW
Published L2R evaluation
SEDD
49.410
41.190
118.740
41.700
72.600
RADD
49.090
38.260
107.780
38.410
63.330
GPT-2
45.040
42.430
138.430
41.600
75.200
σ -GPT †
68.610
57.660
146.870
55.540
90.980
AO-GPT ‡
52.460
42.100
135.960
40.970
71.730
Table 1: Zero-shot perplexity ( ↓ ) on language benchmarks for approximately 125M-parameter models. Published baselines are reproduced from Table 5 of AO-GPT ( Xue et al., 2025 ) . † denotes a model reproduced by the AO-GPT authors, and ‡ denotes AO-GPT trained with an additional 10% of L2R-ordered data. ∗ denotes our AO-GPT baseline trained for 400k updates without language-model EMA or a prescribed L2R training mixture. Published rows follow their original settings and provide numerical context rather than a compute-matched ranking. Fixed-random AR (throughout) uses one presampled random permutation throughout training and inference. Fixed-random AR + curriculum follows OrderGPT’s random-order warm-up and order-use annealing schedule, with a presampled random permutation as its terminal order for subsequent training and inference.
Method
Training data
NLL/joint ↓
Whole ↓
Body ↓
Face-local ↓
Hand-local ↓
Large SimpleBaseline
H3WB
–
131.40
131.60
19.80
44.80
Jointformer
H3WB
–
109.20
103.00
19.80
53.50
Jointformer
H3WB+T3WB
–
84.20
80.10
16.30
44.50
Canonical AR
H3WB
−11.05±0.02
80.83±1.80
74.86±0.59
10.87±0.66
37.41±3.87
Fixed-random AR
H3WB
−10.68±0.09
99.26±11.87
82.69±8.53
13.14±2.68
36.36±1.71
AO-GPT
H3WB
−10.59±0.07
115.23±15.98
89.97±8.40
14.67±2.95
38.18±0.42
Table 2: Performance on H3WB masked-I2D. Internal results are reported as mean ± sample standard deviation across three independently trained model seeds ( n=3 per method). Each run is evaluated on the same 10,000-pose test split. NLL is averaged per joint, and MPJPE values are in millimeters. Published results are taken from Table 4 of Zhu et al. (2023) and provide external numerical context. Lower is better for every metric.
Method
Test NLL/variable ( ×10−3 ) ↓
Q-error P50 ↓
Q-error P99 ↓
Maximum Q-error ↓
Published references
Progressive Sampling
–
1.030±0.010
1.490±0.080
8.860±14.000
Variable Skipping
–
1.090±0.010
2.480±0.290
14.700±7.000
Multi-Order + Skipping
–
1.200±0.240
3.200±0.290
12.600±5.000
Internal evaluations
Canonical AR
325.818±0.020
1.029±0.003
1.459±0.007
2.862±1.137
Table 3: Performance in density modeling and cardinality estimation on CENSUS. Internal results are reported as mean ± sample standard deviation across three independently trained model seeds ( n=3 per method). Test NLL is averaged per variable on the held-out split and displayed in units of 10−3 . Published rows reproduce the reported 10,000-particle results ( Yang et al., 2019 ; Liang et al., 2020 ) . Internal Q-error uses the same 1,000-query workload with 10,000 particles per query. Lower is better, and bold denotes the lowest internal mean.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Seq80
OpenWebText
H3WB
CENSUS
Ordered variables D
80
1,024
133
68
Decoder architecture
4/8/384
12/10/640
4/8/256
2/4/128
Global batch
512
480
160
512
Backbone updates
50,000
400,000
100,000
50,000
Peak learning rate
10−3
6×10−4
3×10−4
3×10−4
Minimum learning rate
10−4
6×10−5
3×10−5
3×10−5
Appendix
Table S1: Backbone settings: architecture denotes decoder layers / heads / hidden width, and batch counts examples per optimizer update.
Setting
Seq80
OpenWebText
H3WB
CENSUS
Examples per attention window
1,024
2,400
1,280
1,024
Updates per window
2
5
8
2
Start distillation
10k
4k
15k
5k
Stop distillation
18k
40k
40k
10k
Annealing complete
32k
60k
60k
20k
Appendix
Table S2: Attention windows and three order-learning milestones.
Figure S1: Loss trajectories: (a) OpenWebText training token NLL ; (b) H3WB validation NLL per joint; (c) CENSUS validation NLL per variable. Vertical guides mark curriculum transitions. The OpenWebText Fixed-random AR + curriculum curve follows the same schedule as OrderGPT. Internal curves represent individual runs and are averaged or smoothed for display.
Figure S2: Post-hoc interpretation of one learned H3WB order. (a) Reveal rank on a representative training-set 2D pose, with face and hand enlargements; darker points appear earlier. (b) Anatomical regions along canonical and learned orders. The pose and connecting lines provide display context, not an input-specific prediction, learned dependency edges, or the evaluation bone set. The pose, canonical identities, and anatomical labels do not enter order learning, direction selection, or model selection.
Table S3: Representative CENSUS neighborhoods in the audited order; positions are one-based. Column names and semantic groups are post-hoc annotations unavailable to the order learner, and the windows need not form a disjoint partition.
Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S;t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S;t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.
Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order inference. For autoregressive language models, which lack a native any-order interface, non-causal abilities such as infilling and next-edit prediction require hand-designed mechanisms. Can we instead design models that natively support any-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any-order training objective naturally offers an any-order prediction interface. This interface, however, does not automatically yield any-order inference. We demonstrate that this interface-inference gap stems from positional uncertainty: fixed-canvas, token-level models may know what semantic component should appear without knowing where to place it. In light of this, we propose two complementary approaches: (1) Insertion-based masked diffusion, building on FlexMDM (Kim et al, 2025), relaxes fixed-position commitments via insertions, enabling generation across non-contiguous regions. (2) Latent-space masked diffusion shifts prediction to coarser semantic segments, enabling search over latent generation orders. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any-order inference behaviors and improve downstream performance. We release our codebase at https://github.com/SeunggeunKimkr/genuine-any-order.
Seunggeun Kim, Jaeyeon Kim, Taekyun Lee +4
University of Texas at Austin · Harvard University
Non-monotonic sequence generation methods, such as masked diffusion models, provide a flexible alternative to left-to-right autoregressive modeling by allowing tokens to be generated in non-fixed and prescribed orders. Despite their practical advantages, most existing non-monotonic models are order-agnostic and rely on a fixed-length grid, limiting their ability to support variable-length generation and adaptive insertion order. In this work, we introduce a probabilistic framework for learning insertion order in variable-length insertion models. We formalize a bijective correspondence between insertion trajectories and permutations, which enables an exact reparameterization of the data likelihood as a sum over permutations. Building on this result, we propose the Insertion Process (IP), a stochastic generative model that jointly learns where to insert, what to insert, and when to terminate, trained via permutation-based variational inference. Unlike prior fixed-canvas approaches, IP natively supports variable-length generation and learns data-driven preferences over insertion orders. Experiments on goal-conditioned planning and molecular string generation demonstrate that learning insertion order improves both modeling quality and generalization in domains without a canonical left-to-right structure.
Yangtian Zhang, Zhe Wang, Arthur Gretton +4
1Yale University · 2Google DeepMind · University College London +1