Conditional Computation

Follow this topic Ask about this topic

Conditional computation studies how to vary a model’s work across inputs, tasks, or runtime budgets instead of executing every layer and component every time. Common approaches use learned gates to route tokens among experts or attention modes, skip layers or regions, choose activations, or allocate additional reasoning when confidence is low. These methods are tested across language models, vision and video, edge inference, and scientific screening, with comparisons focused on accuracy at matched FLOPs, memory, latency, or energy. A central challenge is turning adaptive routing into real hardware savings: gates can add overhead, routing can become unbalanced, and dynamic behavior must be distinguished from static sparsity or parameter changes that do not reduce actual execution.

Latest papers

38 papers10 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Nested Sparsity for Scalable Pretraining and Continual Learning
Oct 22, 2026Tianzhe Chu, Druv Pai, Yi MaConditional Computation Sparse Attention
University of Hong Kong · UC Berkeley
Scene-Adaptive VLA: Efficient Autonomous Driving via Dynamic Layer Routing
Oct 17, 2026Yujia Yang, Quan Yuan, Jiawei Fan +5Efficient VLA Model Inference VLMs for Autonomous Driving
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications
FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task Interference Unified Multimodal Models
Carnegie Mellon University · William & Mary · University of Maryland
QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing
Oct 7, 2026Yueying Li, Zhongle Xie, Ke Chen +1Efficient Transformer Inference Conditional Computation
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models
Oct 6, 2026Jia-Dong ZhangLLM Inference Acceleration Efficient Language Model Inference
LRCC: Generalizing Low-Rank Compression with Conditional Computation
Oct 5, 2026Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti +2LLM Compression Low-Rank Compression
Sapienza University of Rome
CrystalJev: calibrated foundation-model probabilities decide what to compute next in materials discovery
Oct 4, 2026Peng Kang, Yaxuan Sun, Zhen Li +3Materials Discovery Materials Property Prediction
School of Materials Science and Engineering, Beihang University, Beijing 100191, China · National Key Laboratory of Artificial Intelligence for Material Science, Beihang University, Beijing 100191, China
Overcoming Kernel Redundancy for Scaling Logic Gate Networks
Oct 1, 2026Sejin Park, Hongjae Lee, Changwoo Han +1Neural Scaling Laws Conditional Computation
Department of Electrical Engineering Korea University
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Sep 30, 2026Jiaju Wu, Yi Hu, Muhan ZhangTransformer Interpretability Transformer Attention
Institute for Artificial Intelligence, Peking University. · School of Mathematical Sciences, Peking University.
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Language Model Scaling Laws Mixture-of-Experts Models
Tsinghua University · Tianjin University
Learning When to Recur: Token-Adaptive Recursion for Imbalanced Ophthalmic Domain Incremental Learning
Sep 26, 2026Nanxi Yu, Kang Li, Ye Du +3Ophthalmic Image Analysis Domain-Incremental Learning
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Sichuan, China · Department of Biomedical Engineering and Sports Technology, The Hong Kong Polytechnic University, Hong Kong SAR, China · School of Future Technology, South China University of Technology, Guangdong, China +1
MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup
Sep 14, 2026Muchen Li, Leonid Sigal, Renjie LiaoMemory-Augmented Language Models Persistent Memory for Language Models
University of British Columbia · 2Vector Institute for AI · 3Canada CIFAR AI Chair +1
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks
Sep 13, 2026Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias +1Edge Inference Distributed Inference
National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK
An Efficient and Modular Framework for Targeted Harm Mitigation in LLMS
Sep 12, 2026Roberto Campbell, Momin Abbas, Muneeza Azmat +5LLM Safety Alignment Low-Rank Adaptation
University of California, Santa Cruz · IBM Research
The Von-Neumann State-Space Transformer for neural decoding
Aug 25, 2026Morteza SarafyazdTransformer Neural Decoding
BrainCo, Somerville, US.
PromptPath: Prompt-Adaptive Computational Pathways for In-Context Learning
Aug 3, 2026Hangrui Zhang, Feifei Shao, Yawei Luo +6Mixture of Experts In-Context Learning
Zhejiang University, China · University of Nevada, Reno, USA · Guangdong Institute of Intelligence Science and Technology, China
SecondOpinion: Anatomy-Aware Gated Reasoning for Efficient Medical Image Analysis
Aug 3, 2026Siam Tahsin Bhuiyan, Rashedur Rahman, Sefatul Wasi +4Chest X-Ray Classification Medical Image Analysis
Center for Computational & Data Sciences, Independent University, Bangladesh, Dhaka, Bangladesh · Department of Computer Science and Engineering, Independent University, Bangladesh, Dhaka, Bangladesh · Graduate School of Engineering, University of Hyogo, Kobe, Japan
Dynamic Parameterization Is Not Dynamic Inference
Jul 28, 2026Zongfei Li, Yuan-yih Shang, Guozhong LuoEfficient Transformer Inference LLM Auditing
Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators
Jul 24, 2026Afzal Ahmad, Gaoyu Mao, Shoubo Hu +4Structured Sparsity FPGA-Based Neural Network Acceleration
The Hong Kong University of Science and Technology · Huawei Noah’s Ark Lab · The Hong Kong University of Science and Technology (Guangzhou)
SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs
Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1On-Device Language Model Inference Memory-Efficient Inference
Department of Computer Networks, Paderborn University, Paderborn, Germany
Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoning
Jul 12, 2026Sudipto Ghosh, Tanmoy ChakrabortyGroup Relative Policy Optimization Multi-Agent Reasoning
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
Jul 7, 2026Andrii Balashov, Olena PonomarovaSparse Mixture-of-Experts LLM Routing
Ukrainian State University of Science and Technologies
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM Inference Structured Sparsity
University of North Carolina at Chapel Hill · Meta AI
BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving
Jun 7, 2026George Ling, Lijin Yang, Hao Yang +1VLMs for Autonomous Driving Autonomous Driving
Bosch Research
Sigma-Branch: Hierarchical Single-Path Network Reconstruction for Dynamic Inference with Reduced Active Parameters
Jun 7, 2026Kohga Tanaka, Hiroaki NishiEfficient Neural Network Inference Dynamic Neural Networks
Graduate School of Science and Technology, Keio University, Kohoku-ku, Yokohama 223-8522, Japan · Department of System Design, Faculty of Science and Technology, Keio University, Kohoku-ku, Yokohama 223-8522, Japan
Tracing Computation Density in LLMs
May 26, 2026Corentin Kervadec, Iuliia Lysova, Iuri Macocco +2Transformer Interpretability LLM Interpretability
Universitat Pompeu Fabra · ICREA
From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation
May 9, 2026Bohan Li, Shuojue Yang, Baorui Peng +10Action-Conditioned Video Generation Mixture of Experts
1SJTU · 2NUS · 4EIT +5
Linear-Time Global Visual Modeling without Explicit Attention
May 3, 2026Ruize He, Dongchen Han, Gao HuangTransformer Transformer Attention
Tsinghua University
Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study
Apr 19, 2026Qingwei LinEfficient Neural Network Inference Conditional Computation
Independent Researcher
AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
Apr 9, 2026Handong Li, Zikang Liu, Longteng Guo +10Long-Video Understanding Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Alibaba Group Holding Limited +1

Oct 22, 2026J/K move · Enter open · S save

Nested Sparsity for Scalable Pretraining and Continual Learning

Tianzhe Chu, Druv Pai, Yi Ma

University of Hong Kong · UC Berkeley

Conditional Computation Sparse Attention Mixture-of-Experts Models Continual Learning for LLMs Language Model Pretraining

Open paper AskPDF arXiv

Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in each forward pass. In MoE transformers, this sparsity arises in the MoE operator, which generalizes the multi-layer perceptron in the classical transformer. In this work, we explore an alternative mechanism for conditional computation that analogously generalizes the attention operator. The sparsity induced by this mechanism naturally nests (i.e., hierarchically composes) with the sparsity induced by the MoE operator. We show that a transformer employing this nested sparsity mechanism, which we call Nestor, performs better than a conventional MoE transformer at equal compute and equal sparsity in language model pretraining. We also show that Nestor is well-suited for continual pretraining: it can be adapted to longer contexts more effectively than conventional MoEs, as well as adapt more efficiently to new domains while forgetting less on old domains. Furthermore, we demonstrate that these benefits persist as model size, dataset size, and context length grow. Our results demonstrate the potential of alternative conditional computation mechanisms for deep neural network architectures. Code and checkpoints are available on GitHub at https://github.com/Ma-Lab-Berkeley/Nestor .