Conditional Computation

Conditional computation studies how to vary a model’s work across inputs, tasks, or runtime budgets instead of executing every layer and component every time. Common approaches use learned gates to route tokens among experts or attention modes, skip layers or regions, choose activations, or allocate additional reasoning when confidence is low. These methods are tested across language models, vision and video, edge inference, and scientific screening, with comparisons focused on accuracy at matched FLOPs, memory, latency, or energy. A central challenge is turning adaptive routing into real hardware savings: gates can add overhead, routing can become unbalanced, and dynamic behavior must be distinguished from static sparsity or parameter changes that do not reduce actual execution.