cs.DCOct 24, 2026
SaveLLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Organizations: Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.
Abstract
The widespread deployment of foundational Large Language Models (LLM) across cloud datacenters marks a fundamental paradigm shift from stateless deep learning toward distributed, memory-intensive generative workloads. However, serving massive foundation models under stringent Service Level Objectives (SLO) exposes critical physical and system bottlenecks. These challenges include the persistent High Bandwidth Memory (HBM) wall, operational asymmetry between compute-bound prefilling and memory-bandwidth-bound decoding, explosive Key-Value (KV) cache expansion, prohibitive serverless cold starts, and hardware reliability vulnerabilities. Existing literature reviews remain layer-siloed, decoupling low-level accelerator virtualization from cluster orchestration and runtime inference, while largely overlooking defining advancements from 2024 to 2026. To address these limitations, this study presents a comprehensive, cross-layer systematic review guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 framework. We synthesize a holistic taxonomy unifying hardware virtualization, cluster-level scheduling, and runtime inference optimizations, including iteration-level continuous batching, prefill-decoding disaggregation, and hierarchical KV cache offloading. Furthermore, we evaluate resilient cloud-native service architectures, examining scale-to-zero serverless paradigms, fault-tolerant checkpointing against silent data corruption, and hardware-enforced Trusted Execution Environments (TEEs) for multi-tenant isolation. Finally, we formalize the multi-dimensional Pareto trade-offs governing throughput, latency, monetary cost, and confidentiality, outlining a forward-looking research roadmap spanning Compute Express Link (CXL) memory pooling, in-memory computing, and autonomous multi-agent control planes to guide future cloud LLM infrastructure.
Figures & tables
| Ref. | Focus Area | Core Contribution | Reported Limitation |
|---|---|---|---|
| Kachris (2025) | Hardware Architecture | Survey of GPU/FPGA/ASIC accelerators for LLMs; energy-efficiency analysis | Hardware-centric; no link to cloud orchestration or serving systems |
| K. Rostam et al. (2024) | Hardware & System Lifecycle | PRISMA-based systematic review of 65 papers; taxonomy across training, inference, and system serving | Coverage ends December 2023; excludes recent serving/orchestration work |
| Miao et al. (2025) | Inference Serving | MLSys-perspective survey of algorithmic and systems techniques for efficient serving | Emphasizes breadth; limited depth on cloud-specific resource sharing |
| Zhen et al. (2025) | Inference Serving | Hierarchical review: instance-level and cluster-level serving techniques | No comparative experiments; coverage ends April 2025 |
| Jeong and Jeong (2025) | Cloud Orchestration | Taxonomy of reactive/proactive autoscaling policies; lifecycle and security review | General cloud-native scope; not LLM-workload aware |
| Zhang et al. (2025c) | Operational Orchestration | Survey of 183 papers on LLM-driven AIOps across four research questions | Explicitly excludes general LLM limitations (e.g., hallucination) |
Table 1: Comparative summary of surveyed review papers related to LLM architecture, orchestration, and inference in the cloud.
| Survey | Year | Hardware Accel. | Inference Serving | Autoscaling | Orchestration | Reliability | Cloud Native | Future Trends |
|---|---|---|---|---|---|---|---|---|
| Kachris | 2025 | ✓ | partial | ✕ | ✕ | ✕ | partial | ✓ |
| K. Rostam et al. | 2024 | partial | ✓ | partial | partial | partial | ✕ | ✓ |
| Miao et al. | 2025 | partial | ✓ | ✕ | partial | ✕ | partial | ✓ |
| Zhen et al. | 2025 | ✕ | ✓ | partial | ✓ | ✕ | ✓ | ✓ |
| Jeong and Jeong | 2025 | ✕ | ✕ | ✓ | ✓ | partial | ✓ | ✓ |
| Zhang et al. | 2025 | ✕ | ✕ | partial | ✓ | ✕ | partial | ✓ |
Table 2: Comparison with existing review and survey papers on LLM architecture, orchestration, and inference in the cloud.
Figure 1: PRISMA 2020 flow diagram illustrating the study selection process, including identification across four digital libraries, automated deduplication, title and abstract screening, full-text eligibility assessment with explicit exclusion reasons, and final corpus inclusion.
Figure 2: Architectural taxonomy of cloud LLM orchestration and resource scheduling, illustrating the structural hierarchy spanning GPU virtualization abstractions, cluster-level resource scheduling strategies, and high-level LLM serving orchestration paradigms.
Figure 3: Architectural workflow of disaggregated LLM serving and multi-tenant orchestration, illustrating the separation of compute-bound prefill nodes and memory-bound decode nodes, linked by high-speed KV cache interconnects and governed by SLA schedulers, multi-tenant LoRA managers, and tiered memory managers.
| Framework | Primary Domain | Key Architectural Innovation | Primary Optimization Metric | Interconnect / Transfer | Target Deployment |
|---|---|---|---|---|---|
| AntMan ( Xiao et al., 2020 ) | V a | Dynamic memory manager & CUDA API kernel interception | GPU Utilization & QoS | PCIe / Host RAM | Co-located Cluster |
| PipeSwitch ( Bai et al., 2020 ) | V a | Pipelined context switching overlapping weight copy & compute | Model Switching Latency | PCIe Gen3/4 | Multi-Model Serving |
| AlpaServe ( Li et al., 2023 ) | V a | Statistical multiplexing with dynamic model parallelism | Aggregate Throughput | NVLink / PCIe | Model-Parallel Cloud |
| FaST-GShare ( Gu et al., 2023 ) | V a | Spatio-temporal GPU sharing for serverless inference | Function Startup Latency | Shared GPU Memory | Serverless FaaS |
| gShare ( Yang et al., 2026b ) | V a | Aggressive memory allocation for multi-tenant FaaS | OOM Prevention & Latency | Host-to-GPU Memory | Multi-tenant Cloud |
| WAGES ( Wang et al., 2025 ) | V a | Workload-aware GPU sharing and DVFS tuning | Active Energy Consumption | Shared GPU Compute | Serverless LLM |
Table 3: Comparative summary of representative Cloud LLM Orchestration and Resource Scheduling frameworks.
Figure 4: Taxonomy of cloud LLM inference optimization across four architectural pillars: (1) replica and stateful scaling, (2) memory and compute batching, (3) KV cache reuse and side-channel privacy, and (4) multi-tier cost optimization. The taxonomy highlights representative serving systems, underlying execution mechanisms, and cross-cutting co-design challenges.
| System | Approach | Key Mechanism | Reported Gain |
|---|---|---|---|
| PHPA ( Duman and Eken, 2026 ) | Predictive | LLM-based pattern taxonomy and adaptive forecast selection | 37.4% higher prediction accuracy; 96.7% pattern-classification accuracy |
| LSTM-HPA ( Vasumathi et al., 2025 ) | Predictive | LSTM demand forecasting layered on K8s HPA | 28% faster response; 50% less scaling delay; 15% less energy |
| KIS-S ( Zhang et al., 2025b ) | Learned policy | GPU-aware simulator (KISim) + PPO autoscaler | 75.2% reward increase; up to lower P95 latency |
| SkyServe ( Mao et al., 2025 ) | Proactive / Spot | Multi-cloud proactive replica provisioning and preemption-aware migration | Mitigates spot-preemption tail latency; cuts multi-cloud deployment cost |
| SMS Review ( Rhee and Yang, 2026 ) | Conceptual | State-aware reactive/proactive/hybrid taxonomy for stateful microservices | N/A (qualitative architectural taxonomy) |
Table 4: Comparative summary of representative cloud LLM autoscaling systems.
| System | Type | Key Contribution | Headline Result |
| SemShareKV ( Zhao and Mastorakis, 2025 ) | Reuse | Token-level LSH matching for cross-user prefix sharing | Reduced GPU memory footprint |
| SGLang ( Zheng et al., 2024 ) | Reuse | Radix-tree prefix cache management with LRU eviction | Fast multi-turn and prefix KV reuse; low TTFT |
| Pensieve ( Yu et al., 2025 ) | Reuse | Stateful DRAM and GPU memory hierarchy swapping for multi-turn chats | Lower TTFT across turns without pinned GPU memory |
| Shadow in the Cache ( Luo et al., 2026 ) | Attack with Defense | Inversion, collision, injection attacks; KV-Cloak defense | Reconstruction reduced to random noise |
| Semantic Cache Poisoning ( Wu et al., 2026 ) | Attack | Poisoning of production semantic caches (Azure, AWS, and Alibaba) | Confirmed practical across 3 clouds |
| Hardware Side-Channels ( Gao et al., 2025 ) | Attack | Cache-access timing leaks token value and position | 98.7% and 98.0% cosine similarity |
Table 5: Comparative summary of KV-cache reuse and KV-cache security research.
| System | Layer | Key Mechanism | Reported Gain |
|---|---|---|---|
| Splitwise ( Patel et al., 2024 ) | Pipeline | Prefill/decode phase splitting across machines | Up to throughput at 20% lower cost |
| DistServe ( Zhong et al., 2024 ) | Pipeline | SLO-driven prefill/decode disaggregation | Higher goodput under TTFT/TBT SLOs |
| Mooncake ( Qin et al., 2025 ) | Storage tiering | KV-cache-centric global cache across CPU/DRAM/SSD/NIC | 59%–498% higher effective capacity |
| FlashAttention ( Dao et al., 2022 ) | Kernel | IO-aware tiled attention | Up to speedup, exact outputs |
| SmoothQuant ( Xiao et al., 2023 ) | Quantization | Activation-to-weight quantization migration | Up to speedup, memory reduction |
| AWQ ( Lin et al., 2024a ) | Quantization | Activation-aware salient-weight protection | speedup vs. FP16 |
Table 6: Comparative summary of cloud LLM cost-optimization techniques by intervention layer.
| Framework | Category | Key Architectural Innovation | Primary Optimization Metric | Evaluation Platform |
|---|---|---|---|---|
| FlashServe ( Chen, 2026 ) | Cold-Start Mitigation | Tiered DRAM and PCIe snapshotting, Prophet-LSTM prewarming, LoRA multiplexing | 49 cold-start reduction over S3; 3.3 over ServerlessLLM; 32% lower GPU idle cost | Azure Functions trace |
| PipeCo ( Duan et al., 2025 ) | Cold-Start Mitigation | Vertical slice partitioning, attention-based prewarm scheduling, container matching | 62.67%/58.81% E2E latency reduction (CPU/GPU); 65.31% resource reduction | OpenFaaS |
| HydraServe ( Lou et al., 2026 ) | Cold-Start Mitigation | Proactive multi-server model distribution, overlapped cold-start stages, pipeline consolidation | 1.7-4.7 cold-start reduction; 1.43-1.74 SLO attainment | Public cloud |
| PISeL ( Rahimi Jafari et al., 2024 ) | Cold-Start Mitigation | Layer-grouping pipeline of download, deserialize, and execute | 37-63% cold-start speedup; up to 59% peak-memory reduction | PyTorch / TensorFlow (CPU/GPU) |
| FaST-GShare ( Gu et al., 2023 ) | GPU Sharing & Multiplexing | FaaS-oriented spatio-temporal GPU sharing (Manager, Profiler, and Scheduler) | 3.15 throughput; 1.34 GPU util.; 3.13 SM occupancy | OpenFaaS, MLPerf |
| SLINFER ( Xu et al., 2026 ) | GPU Sharing & Multiplexing | Token-level allocation, forward-looking memory scaling, preemption and bin-packing | 47-62% capacity gain (sharing); 86-154% with CPU use | CPU and A100 GPU testbed |
Table 7: Comparative summary of representative serverless architecture frameworks for LLM and DNN inference.
Figure 5: Multi-tier defense-in-depth architecture for cloud LLM dependability. The defense pipeline is organized across four interrelated strata (hardware resilience, statistical reliability, semantic guardrails, and AIOps remediation), progressively intercepting corresponding vulnerability vectors. Representative systems providing algorithmic safeguards for each layer are summarized on the right.
| System | Defense Layer | Key Mechanism | Reported Outcome |
| FT2 ( Sun et al., 2025b ) | Hardware (SDC) | First-token trajectory profiling with layer-selective checksum protection | Comprehensive transient fault recovery at low execution overhead |
| ATTNChecker ( Liang et al., 2025 ) | Hardware (SDC) | Fused mathematical checksum relations tailored for multi-head attention | Near-zero latency overhead fault detection in attention kernels |
| Dr. DNA ( Ma et al., 2024 ) | Hardware (SDC) | Neuron activation distribution profiling and runtime bound checking | Detects soft data corruption without low-level arithmetic hooks |
| RetryTrigger ( Jiao and Yu, 2026 ) | Hardware (SDC) | LightGBM anomaly detection on runtime decoding features and localized retry | Up to 95.33% fewer SDCs at 4.12% average overhead |
| SpotServe ( Miao et al., 2024 ) | Hardware (Instance) | Dynamic reparallelization and lightweight KV-cache checkpointing | Recovers from cloud VM preemption within seconds |
| Beluga ( Yang et al., 2026a ) / Wang et al. ( Wang et al., 2026a ) | Hardware (Statefulness) | CXL shared disaggregated memory pooling and multi-tier storage | Sub-millisecond context failover across host boundaries |
Table 8: Comparative summary of key representative fault-tolerant cloud LLM serving mechanisms.
| Study | Topology | Primary Focus | Key Architectural Mechanism |
|---|---|---|---|
| Qian et al. (2024) | Sequential chat chain | Collaborative software engineering lifecycle | Role-playing communicative reflection across multi-phase SOPs |
| Singh et al. (2024) | Modular workflow patterns | Pattern formalization and workflow benchmarking | Deconstructs workflows into reflection, tool use, planning, and consensus |
| Shih et al. (2026) | Multi-role collaborative | Multi-objective DSE in autonomous driving | Partitioned agent dialogue optimizing Pareto-trade-off search spaces |
| Huang et al. (2025) | Hierarchical planner-worker | Mission planning and telemetry adaptation for autonomous platforms | Closed-loop feedback between physical telemetry and semantic sub-task graphs |
| Lim et al. (2024) | Decentralized industrial | Smart manufacturing and robotic process control | Translates semantic goals into deterministic execution graphs for robots |
| Lee et al. (2025a) | Decentralized collective | Scalable autonomous network management and traffic optimization | Distributed peer agents coordinating real-time routing and load balancing |
Table 9: Comparative summary of key representative agentic workflows and multi-agent systems in cloud environments.
| System / Architecture | Routing Dimension | Target Optimization | Core Dispatching Mechanism |
|---|---|---|---|
| Mooncake ( Qin et al., 2026 ) | Global cache affinity | TTFT reduction and disaggregated cluster throughput | Routes queries to longest prefix match across multi-tier storage pools |
| CacheGen ( Liu et al., 2024 ) | Streaming KV transfer | Cross-node network bandwidth and dynamic load balancing | Quantized and delta-encoded KV cache streaming between workers |
| CacheBlend ( Yao et al., 2025 ) | Composable RAG caching | Cross-document knowledge reuse in dynamic retrieval | Fuses selective KV chunks, recomputing only cross-chunk attention |
| InferSave ( Kim et al., 2025 ) | Hardware instance selection | Total monetary leasing cost with KV offloading | Evaluates PCIe and memory bandwidth to steer requests to optimal VMs |
| DRL based QoS LLM Router ( Yang et al., 2025a ) | Multi-expert edge-cloud | QoS compliance and latency deadline satisfaction | Online optimization matching query complexity to edge versus cloud models |
| Heuristic-Guided DRL LLM Router ( Jain et al., 2025 ) | Mixed-workload balancing | Dual-tier SLO protection (TTFT and ITL) | Isolates prefill-heavy streams from active autoregressive decoding nodes |
Table 10: Comparative summary of representative AI gateway and traffic routing architectures.
| Study | Hardware and Trust Base | Target Workload | Core Mechanism and Protection |
|---|---|---|---|
| Mo et al. (2024) | Multi-TEE taxonomy | Comprehensive machine learning workflows | SoK covering CPU enclaves, GPU TEEs, and side-channel threats |
| Mohan et al. (2024) | Intel TDX and NVIDIA H100 | Enterprise LLM cloud inference | Quantifies PCIe encryption latency and batching bottlenecks in GPU TEEs |
| Chrapek et al. (2025) | Intel TDX and NVIDIA H100 | LLaMA-2 inference scaling up to 70B | Architectural cost-performance analysis of CPU AMX and GPU TEEs |
| Tan et al. (2025) | GPU TEE (Hopper) | Large-scale LLM serving (up to 175B) | Speculative pipelined pre-encryption masking PCIe bus data transfers |
| Wang et al. (2026b) | Arm TrustZone | On-device mobile LLM inference | Pipelined parameter restoration protecting weights with minimal enclave RAM |
| Lin et al. (2026) | CPU TEE and Untrusted GPU | Collaborative LoRA fine-tuning | Keeps adapter states in enclave while offloading tensor operations to GPUs |
Table 11: Comparative summary of representative confidential computing and privacy-preserving AI systems.
Figure 6: A synthesized roadmap illustrating the intersection of core architectural tensions in cloud foundation models and the emerging hardware and algorithmic frontiers they inform.
More in this topic
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 211 papers