Large Language Models for AI Infrastructure
Organizations: Florida State University, Computer Science, Tallahassee, FL, USA · Stevens Institute of Technology, Computer Science, Hoboken, NJ, USA
Abstract
Large language models (LLMs) are increasingly used to design, optimize, and operate the infrastructure behind AI training and serving, yet the literature remains fragmented across hardware design, machine learning systems, electronic design automation, and high-performance computing. We present a comprehensive survey of LLM-assisted AI infrastructure across its lifecycle, from accelerator design and kernel engineering to compilation, deployment, operations, data management, and energy optimization. We organize the literature by the primary resource the LLM manages: artifact-centric methods for computational artifacts, operational-centric methods for deployment context and system state, and resource-centric methods for data and energy. These categories span the lifecycle and cover AI-specific concerns such as distributed-training configuration, inference serving, GPU-cluster scheduling, and data loading. Building on this taxonomy, we conduct a cross-study quantitative synthesis of reported results. Because studies adopt heterogeneous benchmarks, baselines, hardware, and metrics, we compare methods only where evaluation conditions permit and align results along common dimensions: functional correctness, execution performance, throughput, resource and energy consumption, operational outcomes, and optimization cost, keeping each result's native metric, baseline, and setting rather than collapsing them into a single score. This comparison reveals where LLM-driven methods yield consistent gains, how reported gains in workload orchestration depend on the evaluation environment, from simulators to real clusters and production, and where evidence is still lacking. Together, the taxonomy, lifecycle perspective, and quantitative evaluation provide an evidence-grounded reference for LLM-driven engineering of AI infrastructure.
Figures & tables
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Benchmarks and evaluation protocols | |||||||
| RTLLM ( Lu et al., 2023 ) | 2023 | S | Code | RTL, EDA | — | Corr, PPA | P |
| VerilogEval ( Liu et al., 2023b ) | 2023 | S | Code | RTL | — | Corr | P, SFT |
| Revisiting VerilogEval ( Pinckney et al., 2024 ) | 2024 | S | Code | RTL | — | Corr | P |
| Configuration Over Selection ( Shao et al., 2026 ) | 2026 | S | Code | RTL, EDA | — | Corr | P |
| PostEDA-Bench ( Liu et al., 2026b ) | 2026 | C, T | Code, Cfg | EDA | Co | Succ | P, MA |
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Benchmark construction | |||||||
| KernelBench ⋆ ( Ouyang et al., 2025 ) | 2025 | C | Code | GPU | Co, Ex | Corr, Perf | P |
| MultiKernelBench ⋆ ( Wen et al., 2025 ) | 2025 | C | Code | GPU, NPU | — | Corr, Perf | P |
| TritonBench ⋆ ( Li et al., 2025a ) | 2025 | C | Code | GPU | — | Corr, Perf | P |
| AgentKernelArena ⋆ ( Younesian et al., 2026 ) | 2026 | C | Code | GPU | Co, Ex | Corr, Perf | MA |
| CUDABeaver ( Li et al., 2026d ) | 2026 | C, T | Code | GPU | Co, Ex | Corr, Perf | P |
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Compiler optimization and heuristic synthesis | |||||||
| LLM Compiler ( Cummins et al., 2023 ) | 2023 | C | Cfg | CPU | — | Size | SFT |
| AwareCompiler ( Lin et al., 2025 ) | 2025 | C | Cfg | CPU | Co | Size | SFT, RAG |
| CompilerGPT ( Pirkelbauer and Liao, 2025 ) | 2025 | C, T | Code | CPU | Co, Ex | Perf | P |
| Magellan ( Chen et al., 2026a ) | 2026 | C | Code | CPU | Co, Ex | Perf, Size | Evo |
| MOSAIC ⋆ ( Tang et al., 2026 ) | 2026 | C | Code | GPU, CPU | Ex | Perf, Cost | Evo |
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Environment construction and artifact deployment | |||||||
| SpackIt ( Melone et al., 2025 ) | 2025 | C | Cfg | HPC | Co, Ex | Succ | RAG, P |
| Multi-Docker-Eval ( Fu et al., 2025 ) | 2025 | C | Cfg | VM | Ex | Succ, Cost | P, MA |
| DeployBench ( Wang et al., 2026d ) | 2026 | S, C | Plan, Cfg | VM | Ex, Li | Succ | P |
| SWE-World ( Sun et al., 2026a ) | 2026 | C | Code | Sur | LJ | Succ | SFT, RL |
| Kubernetes and network policy configuration | |||||||
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Log parsing and operational analytics | |||||||
| EPIC ( Karimi et al., 2025 ) | 2025 | T | Diag | HPC | Ex | Acc, Cost | MA, SFT |
| HPC Log Parsing ( Karimi et al., 2026 ) | 2026 | T | Diag | Off | — | Acc | SFT |
| Root-cause analysis and support | |||||||
| StateGraph ( Xiang et al., 2025 ) | 2025 | T | Diag | Prod | — | Acc | RAG |
| HPC-LLM ( Shahin and Alsmadi, 2026 ) | 2026 | S | Diag | Off | — | Acc | SFT, RAG |
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Parallelism and sharding configuration | |||||||
| ASAP ⋆ ( Ding et al., 2025 ) | 2025 | T | Cfg | NPU | — | Perf | MA, RAG |
| OptiCo ⋆ ( Chen et al., 2026c ) | 2026 | K, T | Cfg | GPU | Ex | Perf | MA |
| Primus Tuning Agent ⋆ ( Raina et al., 2026 ) | 2026 | K, T | Cfg | GPU, Sim | Ex, Si | Perf | MA |
| Serving and memory policies | |||||||
| ADRS ⋆ ( Cheng et al., 2025b ) | 2025 | C | Policy | Sim | Si | Perf, Cost | Evo |
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Pipeline, query, and contract generation | |||||||
| Data-Pipeline Interface ( Junior et al., 2024 ) | 2024 | S | Plan | — | — | — | P |
| AI Data Contracts ( Bhoite, 2025 ) | 2025 | C | Cfg | DB | St | Acc | SFT |
| DataFlow ⋆ ( Liang et al., 2025 ) | 2025 | S | Code | DB | Ex | Acc | MA |
| StreamLink ( Feng et al., 2025 ) | 2025 | S | Code | DB | St | Corr | SFT |
| Discovery and catalogs | |||||||
| System | Year | Adapt. | |||||
|---|---|---|---|---|---|---|---|
| Energy-aware code generation | |||||||
| Energy-Efficient Code Directions ( Peng et al., 2024 ) | 2024 | C | Code | CPU | Ex | Energy | P |
| Energy-Efficient Code Prompting ( Cappendijk et al., 2024 ) | 2024 | S | Code | CPU | — | Energy | P |
| Energy-Efficient Code Study ( Apsan et al., 2026 ) | 2025 | S | Code | CPU | — | Energy | P |
| LASSI-EE ( Dearing et al., 2025 ) | 2025 | C | Code | GPU | Ex, LJ | Energy | P |
| Power-oriented hardware optimization | |||||||
Large Language Model-Guided Optimization
Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.
See all 660 papers