LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Authors: Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque, Sreezon Das Gupta, Atiar Osman, Md. Abu Obaida Zishan
Organizations: Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.
The widespread deployment of foundational Large Language Models (LLM) across cloud datacenters marks a fundamental paradigm shift from stateless deep learning toward distributed, memory-intensive generative workloads. However, serving massive foundation models under stringent Service Level Objectives (SLO) exposes critical physical and system bottlenecks. These challenges include the persistent High Bandwidth Memory (HBM) wall, operational asymmetry between compute-bound prefilling and memory-bandwidth-bound decoding, explosive Key-Value (KV) cache expansion, prohibitive serverless cold starts, and hardware reliability vulnerabilities. Existing literature reviews remain layer-siloed, decoupling low-level accelerator virtualization from cluster orchestration and runtime inference, while largely overlooking defining advancements from 2024 to 2026. To address these limitations, this study presents a comprehensive, cross-layer systematic review guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 framework. We synthesize a holistic taxonomy unifying hardware virtualization, cluster-level scheduling, and runtime inference optimizations, including iteration-level continuous batching, prefill-decoding disaggregation, and hierarchical KV cache offloading. Furthermore, we evaluate resilient cloud-native service architectures, examining scale-to-zero serverless paradigms, fault-tolerant checkpointing against silent data corruption, and hardware-enforced Trusted Execution Environments (TEEs) for multi-tenant isolation. Finally, we formalize the multi-dimensional Pareto trade-offs governing throughput, latency, monetary cost, and confidentiality, outlining a forward-looking research roadmap spanning Compute Express Link (CXL) memory pooling, in-memory computing, and autonomous multi-agent control planes to guide future cloud LLM infrastructure.
Figures & tables
Ref.
Focus Area
Core Contribution
Reported Limitation
Kachris (2025)
Hardware Architecture
Survey of GPU/FPGA/ASIC accelerators for LLMs; energy-efficiency analysis
Hardware-centric; no link to cloud orchestration or serving systems
K. Rostam et al. (2024)
Hardware & System Lifecycle
PRISMA-based systematic review of 65 papers; taxonomy across training, inference, and system serving
Coverage ends December 2023; excludes recent serving/orchestration work
Miao et al. (2025)
Inference Serving
MLSys-perspective survey of algorithmic and systems techniques for efficient serving
Emphasizes breadth; limited depth on cloud-specific resource sharing
Zhen et al. (2025)
Inference Serving
Hierarchical review: instance-level and cluster-level serving techniques
No comparative experiments; coverage ends April 2025
Jeong and Jeong (2025)
Cloud Orchestration
Taxonomy of reactive/proactive autoscaling policies; lifecycle and security review
General cloud-native scope; not LLM-workload aware
Zhang et al. (2025c)
Operational Orchestration
Survey of 183 papers on LLM-driven AIOps across four research questions
Explicitly excludes general LLM limitations (e.g., hallucination)
Table 1: Comparative summary of surveyed review papers related to LLM architecture, orchestration, and inference in the cloud.
Survey
Year
Hardware Accel.
Inference Serving
Autoscaling
Orchestration
Reliability
Cloud Native
Future Trends
Kachris
2025
✓
partial
✕
✕
✕
partial
✓
K. Rostam et al.
2024
partial
✓
partial
partial
partial
✕
✓
Miao et al.
2025
partial
✓
✕
partial
✕
partial
✓
Zhen et al.
2025
✕
✓
partial
✓
✕
✓
✓
Jeong and Jeong
2025
✕
✕
✓
✓
partial
✓
✓
Zhang et al.
2025
✕
✕
partial
✓
✕
partial
✓
Table 2: Comparison with existing review and survey papers on LLM architecture, orchestration, and inference in the cloud.
Figure 1: PRISMA 2020 flow diagram illustrating the study selection process, including identification across four digital libraries, automated deduplication, title and abstract screening, full-text eligibility assessment with explicit exclusion reasons, and final corpus inclusion.
Figure 2: Architectural taxonomy of cloud LLM orchestration and resource scheduling, illustrating the structural hierarchy spanning GPU virtualization abstractions, cluster-level resource scheduling strategies, and high-level LLM serving orchestration paradigms.
Figure 3: Architectural workflow of disaggregated LLM serving and multi-tenant orchestration, illustrating the separation of compute-bound prefill nodes and memory-bound decode nodes, linked by high-speed KV cache interconnects and governed by SLA schedulers, multi-tenant LoRA managers, and tiered memory managers.
Framework
Primary Domain
Key Architectural Innovation
Primary Optimization Metric
Interconnect / Transfer
Target Deployment
AntMan ( Xiao et al., 2020 )
V a
Dynamic memory manager & CUDA API kernel interception
Layer-grouping pipeline of download, deserialize, and execute
37-63% cold-start speedup; up to 59% peak-memory reduction
PyTorch / TensorFlow (CPU/GPU)
FaST-GShare ( Gu et al., 2023 )
GPU Sharing & Multiplexing
FaaS-oriented spatio-temporal GPU sharing (Manager, Profiler, and Scheduler)
3.15 throughput; 1.34 GPU util.; 3.13 SM occupancy
OpenFaaS, MLPerf
SLINFER ( Xu et al., 2026 )
GPU Sharing & Multiplexing
Token-level allocation, forward-looking memory scaling, preemption and bin-packing
47-62% capacity gain (sharing); 86-154% with CPU use
CPU and A100 GPU testbed
Table 7: Comparative summary of representative serverless architecture frameworks for LLM and DNN inference.
Figure 5: Multi-tier defense-in-depth architecture for cloud LLM dependability. The defense pipeline is organized across four interrelated strata (hardware resilience, statistical reliability, semantic guardrails, and AIOps remediation), progressively intercepting corresponding vulnerability vectors. Representative systems providing algorithmic safeguards for each layer are summarized on the right.
System
Defense Layer
Key Mechanism
Reported Outcome
FT2 ( Sun et al., 2025b )
Hardware (SDC)
First-token trajectory profiling with layer-selective checksum protection
Comprehensive transient fault recovery at low execution overhead
ATTNChecker ( Liang et al., 2025 )
Hardware (SDC)
Fused mathematical checksum relations tailored for multi-head attention
Near-zero latency overhead fault detection in attention kernels
Dr. DNA ( Ma et al., 2024 )
Hardware (SDC)
Neuron activation distribution profiling and runtime bound checking
Detects soft data corruption without low-level arithmetic hooks
RetryTrigger ( Jiao and Yu, 2026 )
Hardware (SDC)
LightGBM anomaly detection on runtime decoding features and localized retry
Up to 95.33% fewer SDCs at 4.12% average overhead
SpotServe ( Miao et al., 2024 )
Hardware (Instance)
Dynamic reparallelization and lightweight KV-cache checkpointing
Recovers from cloud VM preemption within seconds
Beluga ( Yang et al., 2026a ) / Wang et al. ( Wang et al., 2026a )
Hardware (Statefulness)
CXL shared disaggregated memory pooling and multi-tier storage
Sub-millisecond context failover across host boundaries
Mission planning and telemetry adaptation for autonomous platforms
Closed-loop feedback between physical telemetry and semantic sub-task graphs
Lim et al. (2024)
Decentralized industrial
Smart manufacturing and robotic process control
Translates semantic goals into deterministic execution graphs for robots
Lee et al. (2025a)
Decentralized collective
Scalable autonomous network management and traffic optimization
Distributed peer agents coordinating real-time routing and load balancing
Table 9: Comparative summary of key representative agentic workflows and multi-agent systems in cloud environments.
System / Architecture
Routing Dimension
Target Optimization
Core Dispatching Mechanism
Mooncake ( Qin et al., 2026 )
Global cache affinity
TTFT reduction and disaggregated cluster throughput
Routes queries to longest prefix match across multi-tier storage pools
CacheGen ( Liu et al., 2024 )
Streaming KV transfer
Cross-node network bandwidth and dynamic load balancing
Quantized and delta-encoded KV cache streaming between workers
CacheBlend ( Yao et al., 2025 )
Composable RAG caching
Cross-document knowledge reuse in dynamic retrieval
Fuses selective KV chunks, recomputing only cross-chunk attention
InferSave ( Kim et al., 2025 )
Hardware instance selection
Total monetary leasing cost with KV offloading
Evaluates PCIe and memory bandwidth to steer requests to optimal VMs
DRL based QoS LLM Router ( Yang et al., 2025a )
Multi-expert edge-cloud
QoS compliance and latency deadline satisfaction
Online optimization matching query complexity to edge versus cloud models
Heuristic-Guided DRL LLM Router ( Jain et al., 2025 )
Mixed-workload balancing
Dual-tier SLO protection (TTFT and ITL)
Isolates prefill-heavy streams from active autoregressive decoding nodes
Table 10: Comparative summary of representative AI gateway and traffic routing architectures.
Study
Hardware and Trust Base
Target Workload
Core Mechanism and Protection
Mo et al. (2024)
Multi-TEE taxonomy
Comprehensive machine learning workflows
SoK covering CPU enclaves, GPU TEEs, and side-channel threats
Mohan et al. (2024)
Intel TDX and NVIDIA H100
Enterprise LLM cloud inference
Quantifies PCIe encryption latency and batching bottlenecks in GPU TEEs
Chrapek et al. (2025)
Intel TDX and NVIDIA H100
LLaMA-2 inference scaling up to 70B
Architectural cost-performance analysis of CPU AMX and GPU TEEs
Tan et al. (2025)
GPU TEE (Hopper)
Large-scale LLM serving (up to 175B)
Speculative pipelined pre-encryption masking PCIe bus data transfers
Wang et al. (2026b)
Arm TrustZone
On-device mobile LLM inference
Pipelined parameter restoration protecting weights with minimal enclave RAM
Lin et al. (2026)
CPU TEE and Untrusted GPU
Collaborative LoRA fine-tuning
Keeps adapter states in enclave while offloading tensor operations to GPUs
Table 11: Comparative summary of representative confidential computing and privacy-preserving AI systems.
Figure 6: A synthesized roadmap illustrating the intersection of core architectural tensions in cloud foundation models and the emerging hardware and algorithmic frontiers they inform.
Serving large language model (LLM) inference in cloud environments requires jointly optimizing model selection, GPU provisioning, parallelism configuration, and workload routing under latency, accuracy, memory, and budget constraints. While mixed-integer linear programming (MILP) can model this problem, its computational cost limits frequent re-optimization under demand variability. Existing heuristics often optimize individual components separately and may become infeasible when system-wide constraints are enforced. This paper presents a scalable framework for SLO-constrained LLM inference. We formulate the problem as an MILP with a two-phase delay model capturing both prefill and autoregressive decoding under tensor and pipeline parallelism. To solve it efficiently, we develop two constraint-aware heuristics: a Greedy Heuristic (GH) and an Adaptive Greedy Heuristic (AGH). AGH extends GH through multi-start construction, local search, and GPU consolidation. Both methods maintain feasibility through parallelism-aware filtering, cost-based ranking, and adaptive parallelism scaling. Experiments based on the Azure LLM Inference Trace show that GH generates feasible solutions within one second, while AGH achieves near-optimal performance within three seconds and scales to large instances where exact solvers fail to converge. Under out-of-sample stress with up to 1.5x delay and accuracy inflation, AGH degrades gracefully through provisioned headroom, yielding substantially lower cost and SLO violations than cost-minimal MILP solutions. Across synthetic and real Azure workloads, AGH maintains SLO compliance at significantly lower cost than exact MILP solutions. These results demonstrate that high-quality allocations provide substantial robustness to demand variability while enabling rapid adaptation to workload changes.
Jiaming Cheng, Duong Tung Nguyen
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objectives (SLOs) when serving multiple LoRA-adapted services simultaneously. Existing serving frameworks suffer from severe SLO violations due to the computational overhead of LoRA layers and the rigid nature of batch scheduling. To address this, we propose HALO, a scheduling method tailored for LoRA-assisted on-premise LLM deployment. HALO introduces two key innovations: a spatial multiplexing strategy that overlaps Base and LoRA computations by partitioning GPU Streaming Multiprocessors (SMs), and an SLO-aware scheduler that decouples request execution based on "request-level slack." By prioritizing urgent tasks and utilizing idle budget for traffic shaping, HALO significantly mitigates resource contention. Our evaluation demonstrates that HALO minimizes SLO violations while improving throughput compared to state-of-the-art baselines.
Zeshen Zhang, Han Zhao, Weihao Cui +9
Shanghai Jiao Tong University, China · Ant Group, China
LLM inference is increasingly deployed as a core service for enterprise and interactive applications, but sovereignty, compliance, latency and resource-locality requirements can limit the suitability of centralised cloud deployments. Telco operators offer geographically distributed compute and network infrastructure for addressing these constraints. However, existing LLM serving frameworks are designed for tightly coupled data centres and provide limited support for coordinating geographically distributed and heterogeneous resources. We present InferSlice, a reference architecture for Service-Level Objective (SLO)-driven LLM inference over a telco infrastructure, integrating network slicing, GPU partitioning, elastic serving and observability. Led by BT in collaboration with the University of Bristol and NVIDIA, and realised on the JOINER UK national-scale wide-area network (WAN), InferSlice is evaluated on service differentiation across tiers, elastic scale-out, client-edge bandwidth as a bottleneck, and the reach of tensor, pipeline, data parallelism, and prefill/decode disaggregation over the WAN. Our results show that, on a general-purpose telco network, tiers yield distinct latency-throughput regimes, and weight streaming shortens scale-out time. Replication is placement-insensitive but memory-bound; pipeline parallelism tolerates near- and mid-cross-site hops, while tensor parallelism and disaggregation fail beyond a single subnet hop. We share practical insights from operating SLO-driven inference on a telco network, lessons learned and open questions, and hope to encourage further research on distributed and heterogeneous LLM inference systems.
Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar +6