Selecting Optimal Variable Order in Autoregressive Ising Models

Shiba Biswal; Marc Vuffray; Andrey Y. Lokhov

stat.MLFeb 23, 2026

Selecting Optimal Variable Order in Autoregressive Ising Models

Authors: Shiba Biswal, Marc Vuffray, Andrey Y. Lokhov

Organizations: Theoretical Division, Los Alamos National Laboratory, Los Alamos, NM 87545.

Abstract

Autoregressive models enable tractable sampling from learned probability distributions, but their performance critically depends on the variable ordering used in the factorization via complexities of the resulting conditional distributions. We propose to learn the Markov random field describing the underlying data, and use the inferred graphical model structure to construct optimized variable orderings. We illustrate our approach on two-dimensional image-like models where a structure-aware ordering leads to restricted conditioning sets, thereby reducing model complexity. Numerical experiments on Ising models with discrete data demonstrate that graph-informed orderings yield higher-fidelity generated samples compared to naive variable orderings.

Figures & tables

Panel of Figure 1 — Figure 1: An example graph for illustrating the Markov property and the reduction of conditioning sets.

Panel of Figure 2 — Figure 2: Depiction of the three variable order traversals—sequential, checkerboard, and diagonal —on a $5\times 5$ lattice. Node shading indicates the relative selection order, from darker to lighter; identical shading denotes equal preference. Nodes without shading are selected after the shaded nodes and are not distinguished by the ordering. Node labels indicate one specific realization of the traversal.

Panel of Figure 3 — Figure 3: Number of training data samples $M_{l}$ versus sampling error $\varepsilon$ ( 9 ) for both ferro and spin glass Ising models with conditional order $O=6$ . The number of generated samples $M_{s}=10^{5}$ . Error bars are one standard deviation over 20 independent training data sets.

Panel of Figure 4 — Figure 4: Number of generated samples $M_{s}$ versus sampling error $\varepsilon$ ( 9 ) for a ferromagnetic Ising model with different conditional orders $O\in\{2,4,6\}$ , where the conditionals are learned from the ground-truth distribution $p$ . Error bars indicate one standard deviation over 50 independently generated sample sets.

Panel of Figure 5 — Figure 5: Number of training data samples $M_{l}$ versus sampling error $\varepsilon$ ( 9 ) for ferro Ising model with model orders $O\in\{2,4\}$ . The number of generated samples $M_{s}=10^{5}$ . Error bars indicate one standard deviation over 20 independent training data sets.

Panel of Figure 6 — Figure 6: Depiction of two sampling sequences for the D-Wave dataset. The top panel illustrates the sublattice-level selection, while the bottom panel shows the selection order of nodes within each sublattice. Node shading indicates selection order.

Figure 7: Number of generated samples

M_{s}

versus sampling error

\varepsilon

( 9 ) for the D-Wave spin glass Ising model with model order

O=3

, where the conditional distributions are learned from

M_{l}=5\times 10^{5}

. Error bars indicate one standard deviation over 20 independently generated sample sets.

Autoregressive Modeling

Autoregressive modeling predicts sequences step by step from prior context, spanning language, time series, images, audio, and scientific simulations.

See all 85 papers

OrderGPT: Generation Order as a Learnable Modeling Choice
Oct 19, 2026Chenhe Yang, Yuhuan Lyu, Tianyang HuAutoregressive Modeling Autoregressive Generation
SAGA Lab, The Chinese University of Hong Kong, Shenzhen
Attractive and Repulsive Pattern Control in Sequence Generation
Jun 19, 2026Francois PachetMarkov Models Controllable Music Generation
Exact Regular-Constrained Variable-Order Markov Generation via Sparse Context-State Belief Propagation
May 8, 2026François PachetConstrained Generative Modeling Belief Propagation
LIP6, Sorbonne Université · Ynosound

Oct 19, 2026J/K move · Enter open · S save

OrderGPT: Generation Order as a Learnable Modeling Choice

Chenhe Yang, Yuhuan Lyu, Tianyang Hu

SAGA Lab, The Chinese University of Hong Kong, Shenzhen

Autoregressive Modeling Autoregressive Generation

Open paper AskPDF arXiv

Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling. Rather than competing methods, these are the two extreme points of a single axis -- the generation order. For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored. We introduce OrderGPT, which builds on Any-Order GPT to learn a task-level generation order without canonical-order supervision. Our key observation is that predicting variables from randomly sampled reveal prefixes produces recurring attention patterns in the backbone that provide a model-dependent signal of predictive dependencies. We aggregate this signal into a reciprocal dependency graph and use a spectral teacher to derive continuous generation priorities. A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture within a single training run. Training gradually shifts toward the learned order, which is then frozen for subsequent autoregressive training and generation. The same mechanism adapts across modalities: adaptively it recovers near-L2R dependency structure in language, learns anatomically organized factorizations on human pose, and discovers a factorization on tabular data that places statistically and semantically related fields nearby, while improving downstream modeling metrics across all domains. The code is available at https://github.com/Chenhe-yang/OrderGPT/.

Abstract

Autoregressive Modeling

Explore similar work

OrderGPT: Generation Order as a Learnable Modeling Choice

Attractive and Repulsive Pattern Control in Sequence Generation

Exact Regular-Constrained Variable-Order Markov Generation via Sparse Context-State Belief Propagation