Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

Kishan Maharaj; Ashita Saxena; Srikanth Tamilselvam

cs.SEApr 9, 2026

Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

Authors: Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam

Organizations: IBM

Abstract

Autoregressive (AR) language models have driven significant progress in automated software engineering, enabling powerful code generation and assistance systems. However, the next-token prediction paradigm introduces structural limitations for code reasoning, including restricted global planning, challenges in maintaining long-range dependencies, and limited grounding in program execution semantics. Noting the heavy skewness of existing literature towards AR models, we discuss emerging paradigms that could potentially overcome the logic and scaling bottlenecks of next-token prediction by unlocking next-generation architectural capabilities for code intelligence. Specifically, we discuss the potential of Diffusion Models, which generate code via holistic denoising that captures long-range syntactic constraints often missed by AR models. We also discuss Code World Models (CWMs), which simulate execution states to support reasoning, and State Space Models (SSMs), which provide linear-time efficiency for massive contexts. By connecting these developments with findings from cognitive neuroscience, we outline directions for developing "System 2" code generation agents.

Figures & tables

Figure 1: This diagram categorises non-autoregressive paradigms for different code-related tasks. Here, Blue represents diffusion-based approaches, green denotes World Modelling, and red corresponds to State Space paradigms. Based on this, we note that diffusion models remain the most explored non-autoregressive paradigm when compared to World Modelling and State Space Models.

Figure 2: This diagram shows generation dynamics across paradigms. Here, intensity denotes token generation time. Unlike Diffusion’s asynchronous global refinement, Autoregressive models exhibit a steeper intensity gradient than SSMs, reflecting higher sequential decoding latency.

Paradigm	Model / Paper	Task	Benchmark	Metric
Autoregressive	Codex ( Chen, 2021 )	Code generation	HumanEval, APPS	pass@k
Diffusion	CodeFusion ( Singh et al., 2023 )	NL-to-Code generation	CoNaLa, Bash, CF	Template match, Execution match
Diffusion	TreeDiff ( Zeng et al., 2025 )	Code generation (AST-guided)	HumanEval(+), MBPP(+)	pass@1
Hybrid Diffusion	DiffuCoder ( Gong et al., 2025 )	Code generation	HumanEval, MBPP, BigCodeBench, EvalPlus	pass@k
Diffusion	SDLC-Diffusion ( Zhang et al., 2025 )	Software engineering tasks	SDLC benchmark suite	pass@k, workflow metrics
Diffusion	DivoT5 ( Liang et al., 2025 )	Code editing, translation	CodeReview, CodeTrans	EM, BLEU, CodeBLEU

Table 1: Benchmarking practices across major modeling paradigms for code. The table summarizes representative works, tasks, datasets, and metrics used to evaluate autoregressive, diffusion-based, state-space, and world-model approaches.

Code World Models

See all 15 papers

Think Anywhere in Code Generation
Mar 31, 2026Xue Jiang, Tianyu Zhang, Ge Li +8LLM Code Generation Efficient Language Model Reasoning
School of Computer Science, Peking University · Tongyi Lab, Alibaba Group
Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One
Sep 3, 2026Fred Zhangzhi Peng, Kaiwen Zheng, Anru R. ZhangDiffusion Model Distillation One-Step Generative Modeling
Duke University · Tsinghua University
StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
May 12, 2026Hao Wang, Rui Li, Lei Sha +1RL for Language Model Reasoning Execution-Guided Code Generation
Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China +1

Mar 31, 2026J/K move · Enter open · S save

Think Anywhere in Code Generation

Xue Jiang, Tianyu Zhang, Ge Li, Mengyang Liu+7

School of Computer Science, Peking University · Tongyi Lab, Alibaba Group

LLM Code Generation Efficient Language Model Reasoning

Open paper AskPDF arXiv

Recent advances in reasoning Large Language Models (LLMs) have primarily relied on upfront thinking, where reasoning occurs before final answer. However, this approach suffers from critical limitations in code generation, where upfront thinking is often insufficient as problems' full complexity only reveals itself during code implementation. Moreover, it cannot adaptively allocate reasoning effort throughout the code generation process where difficulty varies significantly. In this paper, we propose Think-Anywhere, a novel reasoning mechanism that enables LLMs to invoke thinking on-demand at any token position during code generation. We achieve Think-Anywhere by first teaching LLMs to imitate the reasoning patterns through cold-start training, then leveraging outcome-based RL rewards to drive the model's autonomous exploration of when and where to invoke reasoning. Extensive experiments on four mainstream code generation benchmarks (i.e., LeetCode, LiveCodeBench, HumanEval, and MBPP) show that Think-Anywhere achieves state-of-the-art performance over both existing reasoning methods and recent post-training approaches, while demonstrating consistent generalization across diverse LLMs. Our analysis further reveals that Think-Anywhere enables the model to adaptively invoke reasoning at high-entropy positions, providing enhanced interpretability.

Abstract

Code World Models

Explore similar work

Think Anywhere in Code Generation

Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning