Efficient Inference

Efficient inference studies how to get useful model outputs with less computation, memory, time, or energy, which matters for large-scale serving and devices with limited resources. Approaches include quantization and distillation, caching and reuse, sparse or low-rank computation, early exits, amortized inference, and adaptive allocation of test-time compute. Systems work complements these methods with batching, scheduling, compilation, and hardware-aware execution, while probabilistic methods seek faster posterior or marginal estimates without losing calibration or exactness. The central tension is reducing cost without undermining accuracy, uncertainty estimates, or robustness, and results can depend strongly on the workload and hardware used for evaluation.