We present an industry experience report on three years of operating an event-driven cloud infrastructure for continuous machine learning training in automotive manufacturing. Our system orchestrates GPU-accelerated training of product-specialized model pairs, a physics prediction model and a reinforcement-learning control policy, across multiple plants, coordinating long-running GPU workloads triggered by manufacturing events. The architecture combines Amazon ECS with EC2 GPU capacity providers, SQS-based messaging with dead-letter queues, and an admission-controlled Lambda dispatcher that enforces cluster concurrency limits. A Conductor orchestrator on ECS Fargate initiates dependency-aware retraining chains on a weekly schedule. The entire infrastructure is codified in modular Terraform with multi-account separation. From 40000+ production training jobs we report a 72-78% cost reduction versus always-on GPU infrastructure. A discrete-event simulation confirms that admission control is necessary (naive dispatch loses 65% of jobs) and that queue-draining matches AWS Step Functions latency while eliminating per-job startup overhead. We provide lessons learned and release the simulator and Terraform module skeletons as open-source artifacts.
Figures & tables
Fig. 1: End-to-end manufacturing ML workflow per product per plant line. Triggering events initiate an LPP training request with available new data. The LPP physics model trains on GPU and backtests on CPU (Fargate); on pass, the LCP control policy trains on GPU and backtests on CPU; on pass, the LCP deploys to the edge controller. Each transition is decoupled through a durable SQS queue (Section III-D ).
Account
Resources
Shared Services
Terraform state (S3 + DynamoDB locking). No compute resources.
Tooling (CI/CD)
CodePipeline, CodeBuild (build + deploy), ECR base images, VPC, CodeArtifact
TABLE II: Compute Tier Selection per Pipeline Stage
Fig. 2: AWS-native realization of the workflow in Fig. 1 . EventBridge triggers the Conductor (Fargate), which submits to MlopsEventsQueue . The mlopsCoordinator Lambda enforces concurrency limits before launching GPU tasks. Each stage communicates asynchronously through SQS queues. DLQs capture failures at every stage.
Fig. 3: Admission-controlled dispatcher logic (runs in Lambda, < 1 s). The Lambda parses each SQS message, queries active ECS tasks, and either dispatches a new GPU task or forwards to the task queue for deferred draining by already-running tasks (Fig. 4 ). In both cases, the original message is auto-deleted from the source queue.
Fig. 4: ECS task handler wrapper flow. Phase 1 processes the initial work item from CLI arguments. Phase 2 drains the task message queue via long-polling until 3 consecutive empty polls (60s idle), maximizing GPU utilization per launched instance.
Fig. 5: Multi-account CI/CD. CodeBuild assumes a cross-account LambdaDeploymentRole (with external ID) to update the Coordinator Lambda. ECS tasks pull :latest from ECR on subsequent runs.
Fig. 6: Micro-benchmark across three strategies on identical 52-week workload (2,077 requests, seed-fixed). Left: Naive dispatch loses 65% of jobs to placement failures; both Step Functions and our pattern lose zero. Right: Step Functions and our pattern achieve similar median latency; naive is dominated by retry backoff.
Criterion
SageMaker Pipelines
Step Functions + ECS
Apache Airflow (MWAA)
Self-Managed K8s (EKS)
Our Design (SQS + Lambda + ECS)
GPU startup latency
3–5 min (instance per job)
Same as ours (shared cluster)
Same as ours (external compute)
< 10 s (warm node pool)
< 30 s (pre-warmed ASG)
Cost overhead
Per-job mgmt fee + instance cost
$0.025/1K state transitions
$0.49/hr (smallest MWAA env)
Cluster mgmt + node idle
SQS + Lambda (negligible)
Max job duration
Unlimited
1 year (execution limit)
Unlimited
Unlimited
Unlimited
GPU concurrency control
Built-in parallelism config
Map state parallelism
Pool slots (scheduler-level)
K8s quotas / gang scheduling
Custom (Lambda + ListTasks + ASG-aligned)
Fault isolation
Pipeline-level retry
Catch/retry state machine
Task-level retry + SLA alerts
Pod restart + PDBs
SQS DLQ + visibility timeout
Scale-to-zero compute
No (always-on instances)
Yes (Fargate)
No (scheduler always-on)
No (control plane always-on)
Yes (full scale-to-zero)
TABLE IV: Architecture Alternatives Comparison (Qualitative; quantitative comparison vs. Step Functions in Fig. 6 )
Fig. 7: Monthly per-facility cost comparison. Always-On : dedicated GPU instance 24/7. On-Demand : event-driven, all on-demand GPU pricing. With Spot : event-driven, spot instances for LPP training (LCP requires on-demand).
Fig. 8: GPU utilization over a representative 4-week period. Event-driven: 94% during bursts, 0% idle (scale-to-zero). Always-on: ∼ 50% peak, < 15% baseline, paying for idle GPU 80% of the time.
Fig. 9: DLQ message classification. Data quality issues dominate (65%), addressed by upstream ETL fixes and message replay. GPU OOM (18%) and EC2 capacity (12%) auto-resolve via SQS retry; visibility-timeout cases (5%) prompted Terraform tuning.