State-Aware Activation Recomputation with Replay Contracts and Selective Preservation
Organizations: 1Peking University · 2Beihang University.
Abstract
Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a different computation. We make this state explicit. A versioned replay contract specifies what replay must restore, and a joint planner chooses, per segment, between activation retention, whole-buffer snapshots and selective-region preservation, charging each option its measured memory and recovery cost. Independent derivative and byte-level checks validate the contract, and a Torch/JAX cohort retains all 5,900 execution records, including matches, rejections and divergences. On frozen sparse-write H100 workloads, each measured over fifty fresh processes, joint planning reduces paired step time by 18.16% against a correctly costed eager/whole planner at a 44 GiB budget (approximate 97.5% ratio interval ). At 32 GiB, where the eager/whole planner has no feasible static policy, joint planning is 4.98% faster than fixed regions at 29.94 rather than 26.20 GiB. A complete CIFAR-100 dense-queue application runs seven methods for 100 encoder and 25 probe epochs at six training seeds, with byte-identical test scores within each seed. There, joint planning uses 22.41% more allocated memory than whole snapshots, with a process-time geometric-mean ratio of 1.0080 (seed range ). Together, the results map where selective preservation pays off: sparse writes with large activations benefit, while in the dense-queue workload whole snapshots use less allocated memory.
Figures & tables
| Engine | Retained state | Recovery for backward | Supported boundary |
|---|---|---|---|
| Eager retention | Full pre-operation states | Read the saved prestate | Explicit finite storage and recorded operation descriptors |
| Whole checkpoint | Periodic full-state anchors and recorded random inputs | Replay from the preceding anchor to the required operation | Replay uses the original noise/mask records |
| Read-set checkpoint | Historical values read by each operation | Read the retained operands directly | Read indices are declared by the finite program |
| Region journal | Old and new values of overwritten destinations | Undo writes in reverse order; redo writes to restore the final state | Destinations and write order are explicitly recorded |
| Guarded region | The selected journal or whole-anchor representation | Use whole checkpoints when journal payload exceeds anchor payload; otherwise undo/redo | A byte-count selection rule, rather than measured latency optimization |
| Backend / execution | M | R | D |
|---|---|---|---|
| Torch functional reference | 500 | 0 | 0 |
| Torch native eager | 400 | 100 | 0 |
| Torch reentrant | 200 | 100 | 200 |
| Torch nonreentrant | 200 | 100 | 200 |
| Torch reentrant, RNG unpreserved | 0 | 100 | 400 |
| Torch nonreentrant, RNG unpreserved | 0 | 100 | 400 |
| Validation quantity | Result |
|---|---|
| Independent semantic programs | 1,200 |
| Complex-step coordinates | 4,544 |
| Maximum complex-step error | |
| Joint-alias programs | 300 |
| Supported joint recoveries | 1,500/1,500 |
| Maximum joint gradient error |
| Activation / writes | Eager | Whole | Regions | Functional | Generic | Joint |
|---|---|---|---|---|---|---|
| Small / sparse | 1.22 / 8.73 | 18.21 / 65.39 | 1.35 / 25.38 | 18.21 / 69.75 | 1.22 / 8.69 | 1.22 / 8.70 |
| Small / dense | 2.38 / 38.37 | 19.38 / 126.29 | 19.38 / 140.10 | 19.38 / 130.49 | 2.38 / 38.40 | 2.38 / 38.43 |
| Large / sparse | 46.06 / 2882 | 42.06 / 3937 | 26.20 / 3908 | 42.06 / 3960 | 42.81 / 3750 | 29.94 / 3698 |
| Large / dense | 46.08 / 2921 | 42.33 / 4008 | 42.33 / 4049 | 42.33 / 4043 | 43.08 / 3828 | Infeasible |
| Cell | Minimum predicted GiB | Fits 32 GiB? |
|---|---|---|
| Small sparse | 1.2211 | Yes |
| Small dense | 2.3887 | Yes |
| Large sparse | 40.0631 | No |
| Large dense | 40.0786 | No |
| Method | Peak GiB | Step ms | Fits? |
|---|---|---|---|
| Eager | 46.063 | 2865.83 | No |
| Whole snapshots | 42.063 | 3919.70 | Yes |
| Fixed regions | 26.200 | 3867.19 | Yes |
| Functional | 42.063 | 3932.56 | Yes |
| State-agnostic planner | 45.313 | 3068.25 | No |
| State-aware eager/whole | 42.813 | 3735.42 | Yes |
| 44 GiB cohort | 32 GiB cohort | |||||
|---|---|---|---|---|---|---|
| Method | Step ms | Peak GiB | Fits/50 | Step ms | Peak GiB | Fits/50 |
| Eager | 2869.65 | 46.063 | 0 | 2878.70 | 46.063 | 0 |
| Whole snapshots | 3940.38 | 42.063 | 50 | 3945.99 | 42.063 | 0 |
| Fixed regions | 3889.79 | 26.200 | 50 | 3898.82 | 26.200 | 50 |
| Functional | 3942.81 | 42.063 | 50 | 3952.28 | 42.063 | 0 |
| State-agnostic planner | 3070.72 | 45.313 | 0 | 3744.73 | 42.813 | 0 |
| Workload | Step time | Peak allocated memory |
|---|---|---|
| Queue | .3954 [.3905,.4020] | .05055 [.05055,.05055] |
| Prototype | .6085 [.6078,.6091] | .15035 [.15030,.15039] |
| Recurrence | .4895 [.4890,.4900] | .06476 [.06476,.06476] |
| Method | Allocated MiB | Reserved MiB | Mean encoder seconds | Min–max seconds |
|---|---|---|---|---|
| Eager | 1389.35 | 1924.00 | 1857.47 | 1844.03–1866.87 |
| Whole | 1004.62 | 1410.00 | 1841.10 | 1821.24–1864.30 |
| Regions | 1004.74 | 1410.00 | 1842.40 | 1834.85–1854.06 |
| Functional | 1004.62 | 1410.00 | 1847.51 | 1837.14–1862.65 |
| Deferred | 1389.41 | 1924.00 | 1842.81 | 1833.59–1850.69 |
| Agnostic | 1005.48 | 1410.00 | 1856.22 | 1841.82–1882.82 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Workload | MiB | Write (%) | Live | Peak/whole | Time/whole | Peak/eager | Time/eager |
|---|---|---|---|---|---|---|---|
| Queue | 64 | 0.1 | 1 | 0.669 | 1.134 | 1.002 | 2.263 |
| Queue | 64 | 0.1 | 64 | 0.099 | 1.189 | 1.353 | 3.292 |
| Queue | 64 | 100 | 1 | 1.000 | 1.405 | 1.497 | 2.752 |
| Queue | 64 | 100 | 64 | 1.000 | 1.199 | 11.660 | 3.273 |
| Queue | 1024 | 0.1 | 1 | 0.376 | 0.443 | 1.002 | 2.227 |
| Queue | 1024 | 0.1 | 64 | 0.051 | 0.390 | 2.612 | 2.965 |
Memory-Efficient Training
Memory-efficient training reduces the memory needed to optimize neural networks by compressing, recomputing, or relocating training state.
See all 100 papers