Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing
Organizations: SVKM’s NMIMS (Deemed to be University)
Abstract
Elastic Compute Cloud (EC2) Spot is 60% to 90% cheaper than On-Demand but can be reclaimed on just a 2-minute notice; for expensive multi-node training this loss can be severe, with one reclaim costing hours of synchronous progress. We build Argus, a Kubernetes operator, and ask empirically, on a CIFAR-10 testbed, when predicting interruptions beats simple checkpointing. Argus on real EKS survives a real Spot drain with a graceful SIGTERM checkpoint, resuming from epoch 8 and losing only the in-progress epoch. Alongside, we further find that in an 80-trial benchmark, the reactive-on-notice degrades toward no protection once interruption outpaces the fixed 2-minute notice, and predictive wasted compute is driven to zero, but with an oversized fixed lead it over-migrates so severely that at the fastest rate only one of five runs completes, while periodic is a strong ML-free baseline. A lead-time sweep turns the lead prediction into a guideline where a small lead suffices for zero waste, but excess lead is wasteful. The predictor built is advisory (a proxy label); real interruption labels and large-model-scale validation are future work.
Figures & tables
| Arm | Compl. | Wasted (s) | Makespan (s) | Ckpts |
|---|---|---|---|---|
| No-protection | 1.0 | 361.8 | 0.0 | |
| Reactive-on-notice | 1.0 | 328.0 | 4.8 | |
| Periodic | 1.0 | 158.8 | 24.0 | |
| Predictive (Argus) | 0.2 | 353.2 | 221.6 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Time (UTC) | Event |
|---|---|
| 12:26:18 | NTH receives EC2 Spot Instance Interruption Warning |
| 12:26:19 | Drain evicts pod cifar10-test (graceful SIGTERM) |
| 12:26:19 | SIGTERM handler writes checkpoint (epoch 8) to S3 |
| 12:26:25 | Node cordoned and drained in 10 s |
| 75 s later | Replacement pod on healthy node: “Resuming from epoch 8” |
| Lead (s) | Wasted (s) | Makespan (s) | Checkpoints |
|---|---|---|---|
| 2 | 0.0 | 73.1 | 5.4 |
| 5 | 0.0 | 76.2 | 7.4 |
| 10 | 0.0 | 78.0 | 8.8 |
| 15 | 0.0 | 83.3 | 12.6 |
| 20 | 0.0 | 87.0 | 15.2 |
| 30 | 0.0 | 96.7 | 21.8 |