cs.AIOct 31, 2026

State-Aware Activation Recomputation with Replay Contracts and Selective Preservation

Authors: Annan Zhang, Shengjie Song, Wenshuai Yao, Lihao Zhang, Tianmiao

Organizations: 1Peking University · 2Beihang University.

Abstract

Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a different computation. We make this state explicit. A versioned replay contract specifies what replay must restore, and a joint planner chooses, per segment, between activation retention, whole-buffer snapshots and selective-region preservation, charging each option its measured memory and recovery cost. Independent derivative and byte-level checks validate the contract, and a Torch/JAX cohort retains all 5,900 execution records, including matches, rejections and divergences. On frozen sparse-write H100 workloads, each measured over fifty fresh processes, joint planning reduces paired step time by 18.16% against a correctly costed eager/whole planner at a 44 GiB budget (approximate 97.5% ratio interval [.8172,.8195][.8172,.8195]). At 32 GiB, where the eager/whole planner has no feasible static policy, joint planning is 4.98% faster than fixed regions at 29.94 rather than 26.20 GiB. A complete CIFAR-100 dense-queue application runs seven methods for 100 encoder and 25 probe epochs at six training seeds, with byte-identical test scores within each seed. There, joint planning uses 22.41% more allocated memory than whole snapshots, with a process-time geometric-mean ratio of 1.0080 (seed range [.9845,1.0191][.9845,1.0191]). Together, the results map where selective preservation pays off: sparse writes with large activations benefit, while in the dense-queue workload whole snapshots use less allocated memory.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation CompressionMemory Compression

  2. Manufacturing Differential Sparsity: Planner-Hardware Co-Design for Efficient World Model Planning

    Nov 17, 2026Wenbo Zhu, Paul Yi-Chia Chen, Yuanhan Li +2Hardware-Software Co-DesignWorld Model-Based Planning