Continuous deployment to large scale production systems creates a tension between release velocity and reliability. Every change is a potential reliability incident, yet every delay is a missed opportunity. This paper describes the deployment time health check infrastructure that Meta uses to mediate this tension across thousands of heterogeneous services. We summarize the architecture of this prevention based distributed system's service called Service Health Checker, explain how check authors compose templated metric queries, thresholds, and workflow predicates; and discuss how the system is integrated with tiered and phased rollouts so that regressions trigger automatic rollback. We then describe the operational problems that emerged at scale, such as noise, alert fatigue, drift, and uncovered regressions, and the program of measurement, tooling, and improved defaults we deployed to address them. We close with lessons learned from years of operating deployment health checks at Meta, and the directions we are exploring next, including AI assisted health check tuning. Index Terms: deployment safety, continuous deployment, monitoring, software reliability, release engineering, software reliability engineering, AIOps, anomaly detection
Figures & tables
Fig. 1: Progression of health checks throughout the deployment lifecycle. At each phase or tier of deployment, SHC monitors metrics for the hosts where changes have been rolled out for a predefined period (i.e., bake time) and runs health checks to confirm the health of the system post-deployment. On a failure verdict, the pipeline rolls back or pauses depending on the deployment strategy, and notifies the service owner.
Fig. 2: Anatomy of a health check. A templated metric query yields a time series for each of the test and control cohorts. The shaded region represents the time when the change is being deployed. The health check can define both static bounds, outside of which the series is considered unhealthy, and comparison thresholds, which compare the test signal during the test time window against either the test signal during a control time window (in the case of single-host services) or the control signal during the test window (in the case of multi-host services).
Fig. 3: Backtesting tool that enables service owners to test a candidate check against historical data. The owner can see how the proposed definition would have classified past incidents and past noise before the check is deployed.
Fig. 4: Metric targeting. Templated cohorts let SHC evaluate the same health check at the right granularity for each deployment phase, from a small canary slice up to a regional cohort.