We present an industry experience report on three years of operating an event-driven cloud infrastructure for continuous machine learning training in automotive manufacturing. Our system orchestrates GPU-accelerated training of product-specialized model pairs, a physics prediction model and a reinforcement-learning control policy, across multiple plants, coordinating long-running GPU workloads triggered by manufacturing events. The architecture combines Amazon ECS with EC2 GPU capacity providers, SQS-based messaging with dead-letter queues, and an admission-controlled Lambda dispatcher that enforces cluster concurrency limits. A Conductor orchestrator on ECS Fargate initiates dependency-aware retraining chains on a weekly schedule. The entire infrastructure is codified in modular Terraform with multi-account separation. From 40000+ production training jobs we report a 72-78% cost reduction versus always-on GPU infrastructure. A discrete-event simulation confirms that admission control is necessary (naive dispatch loses 65% of jobs) and that queue-draining matches AWS Step Functions latency while eliminating per-job startup overhead. We provide lessons learned and release the simulator and Terraform module skeletons as open-source artifacts.
Figures & tables
Fig. 1: End-to-end manufacturing ML workflow per product per plant line. Triggering events initiate an LPP training request with available new data. The LPP physics model trains on GPU and backtests on CPU (Fargate); on pass, the LCP control policy trains on GPU and backtests on CPU; on pass, the LCP deploys to the edge controller. Each transition is decoupled through a durable SQS queue (Section III-D ).
Account
Resources
Shared Services
Terraform state (S3 + DynamoDB locking). No compute resources.
Tooling (CI/CD)
CodePipeline, CodeBuild (build + deploy), ECR base images, VPC, CodeArtifact
TABLE II: Compute Tier Selection per Pipeline Stage
Fig. 2: AWS-native realization of the workflow in Fig. 1 . EventBridge triggers the Conductor (Fargate), which submits to MlopsEventsQueue . The mlopsCoordinator Lambda enforces concurrency limits before launching GPU tasks. Each stage communicates asynchronously through SQS queues. DLQs capture failures at every stage.
Fig. 3: Admission-controlled dispatcher logic (runs in Lambda, < 1 s). The Lambda parses each SQS message, queries active ECS tasks, and either dispatches a new GPU task or forwards to the task queue for deferred draining by already-running tasks (Fig. 4 ). In both cases, the original message is auto-deleted from the source queue.
Fig. 4: ECS task handler wrapper flow. Phase 1 processes the initial work item from CLI arguments. Phase 2 drains the task message queue via long-polling until 3 consecutive empty polls (60s idle), maximizing GPU utilization per launched instance.
Fig. 5: Multi-account CI/CD. CodeBuild assumes a cross-account LambdaDeploymentRole (with external ID) to update the Coordinator Lambda. ECS tasks pull :latest from ECR on subsequent runs.
Fig. 6: Micro-benchmark across three strategies on identical 52-week workload (2,077 requests, seed-fixed). Left: Naive dispatch loses 65% of jobs to placement failures; both Step Functions and our pattern lose zero. Right: Step Functions and our pattern achieve similar median latency; naive is dominated by retry backoff.
Criterion
SageMaker Pipelines
Step Functions + ECS
Apache Airflow (MWAA)
Self-Managed K8s (EKS)
Our Design (SQS + Lambda + ECS)
GPU startup latency
3–5 min (instance per job)
Same as ours (shared cluster)
Same as ours (external compute)
< 10 s (warm node pool)
< 30 s (pre-warmed ASG)
Cost overhead
Per-job mgmt fee + instance cost
$0.025/1K state transitions
$0.49/hr (smallest MWAA env)
Cluster mgmt + node idle
SQS + Lambda (negligible)
Max job duration
Unlimited
1 year (execution limit)
Unlimited
Unlimited
Unlimited
GPU concurrency control
Built-in parallelism config
Map state parallelism
Pool slots (scheduler-level)
K8s quotas / gang scheduling
Custom (Lambda + ListTasks + ASG-aligned)
Fault isolation
Pipeline-level retry
Catch/retry state machine
Task-level retry + SLA alerts
Pod restart + PDBs
SQS DLQ + visibility timeout
Scale-to-zero compute
No (always-on instances)
Yes (Fargate)
No (scheduler always-on)
No (control plane always-on)
Yes (full scale-to-zero)
TABLE IV: Architecture Alternatives Comparison (Qualitative; quantitative comparison vs. Step Functions in Fig. 6 )
Fig. 7: Monthly per-facility cost comparison. Always-On : dedicated GPU instance 24/7. On-Demand : event-driven, all on-demand GPU pricing. With Spot : event-driven, spot instances for LPP training (LCP requires on-demand).
Fig. 8: GPU utilization over a representative 4-week period. Event-driven: 94% during bursts, 0% idle (scale-to-zero). Always-on: ∼ 50% peak, < 15% baseline, paying for idle GPU 80% of the time.
Fig. 9: DLQ message classification. Data quality issues dominate (65%), addressed by upstream ETL fixes and message replay. GPU OOM (18%) and EC2 capacity (12%) auto-resolve via SQS retry; visibility-timeout cases (5%) prompted Terraform tuning.
Elastic Compute Cloud (EC2) Spot is 60% to 90% cheaper than On-Demand but can be reclaimed on just a 2-minute notice; for expensive multi-node training this loss can be severe, with one reclaim costing hours of synchronous progress. We build Argus, a Kubernetes operator, and ask empirically, on a CIFAR-10 testbed, when predicting interruptions beats simple checkpointing. Argus on real EKS survives a real Spot drain with a graceful SIGTERM checkpoint, resuming from epoch 8 and losing only the in-progress epoch. Alongside, we further find that in an 80-trial benchmark, the reactive-on-notice degrades toward no protection once interruption outpaces the fixed 2-minute notice, and predictive wasted compute is driven to zero, but with an oversized fixed lead it over-migrates so severely that at the fastest rate only one of five runs completes, while periodic is a strong ML-free baseline. A lead-time sweep turns the lead prediction into a guideline where a small lead suffices for zero waste, but excess lead is wasteful. The predictor built is advisory (a proxy label); real interruption labels and large-model-scale validation are future work.
Industrial IoT sensors produce continuous, high-volume data streams that predictive maintenance needs to analyse in real time. Cloud-centric architectures send all raw telemetry to central servers, which costs latency, bandwidth and resilience. This paper presents a hybrid cloud-edge architecture for manufacturing predictive maintenance built on AWS IoT Core and AWS IoT Greengrass v2, with local machine learning inference at the edge and selective cloud synchronization. We deploy two configurations on real AWS infrastructure and compare them: a cloud-only baseline (Configuration A) and the proposed hybrid edge architecture (Configuration B). We measure end-to-end inference latency, bandwidth consumption and inference accuracy. Configuration B cuts cloud-bound bandwidth by 86.8% (63.6 KB to 7.2 KB per 200-reading window) and decision latency by 256x (23.1 ms cloud round trip to 0.09 ms local inference). At the edge, with no cloud involvement, it keeps an anomaly-class F1 score of 0.84 and 91% recall. The evaluation also instruments system uptime and Stream Manager queuing under cloud disconnection; the quantitative disconnection experiment is left as planned work. Our review of the literature points to four gaps that this work addresses: the lack of empirically validated Greengrass v2 deployments for predictive maintenance, unquantified bandwidth savings from selective edge synchronization, unmeasured resilience under cloud disconnection, and the open MLOps feedback loop at the edge.
Multi-agent LLM systems spend additional computation on reasoning and collaboration, but shared serving queues can prevent that computation from producing answers before a deadline. This creates a coupled decision: the reasoning workflow determines the load, while changing load determines which reasoning choices remain affordable. We introduce INFRAMIND, a hierarchical policy that adapts workflow planning and execution to serving conditions. At arrival, a planner selects the collaboration graph using task information, load and the time budget. Before each agent call, an executor jointly selects the model and reasoning strategy using refreshed telemetry and remaining time. A shared latency penalty connects the two policies. Our analysis characterizes why observing load matters when congestion reverses the preferred computation. Across five benchmarks on a shared five-model deployment, INFRAMIND improves low-load accuracy by up to 7.6 percentage points over the strongest evaluated baseline. On 500 MATH questions with matched EDF scheduling and per-request deadlines of 10-300 s, it increases correct, on-time answers from 20.4% to 33.8%. With 15 models and Azure timings replayed on the same benchmark and deadline tiers, this fraction rises from 11.6% to 54.8% over MasRouter. These results support adapting reasoning decisions to the serving capacity available throughout a workflow.