GPU design space exploration (DSE) for modern AI workloads, such as Large-Language Model (LLM) inference, is challenging because of GPUs' vast, multi-modal design spaces, high simulation costs, and complex design optimization objectives (e.g. performance, power and area trade-offs). Existing automated DSE methods are often prohibitively expensive, either requiring an excessive number of exploration samples or depending on intricate, manually crafted analyses of interdependent critical paths guided by human heuristics. We present LUMINA, an LLM-driven GPU architecture exploration framework that leverage AI to enhance the DSE efficiency and efficacy for GPUs. LUMINA extracts architectural knowledge from simulator code and performs sensitivity studies to automatically compose DSE rules,which are auto-corrected during exploration. A core component of LUMINA is a DSE Benchmark that comprehensively evaluates and enhances LLMs' capabilities across three fundamental skills required for architecture optimization, which provides a principled and reproducible basis for model selection and ensuring consistent architectural reasoning. In the design space with 4.7 million possible samples, LUMINA identifies 6 designs of better performance and area than an A100 GPU efficiently, using only 20 steps via LLM-assisted bottleneck analysis. In comparison, LUMINA achieves 17.5x higher than design space exploration efficiency, and 32.9% better designs (i.e. Pareto Hypervolume) than Machine-Learning baselines, showcasing its ability to deliver high-quality design guidance with minimal search cost.
Figures & tables
Figure 1. Design space visulizatin for a GPT3-175B ( Brown et al., 2020 ) inference workload via roofline model ( Williams et al., 2009 ) . Each sampled architecture is embedded via Principal Component Analysis ( Gewers et al., 2018 ) into two dimensions; distributions are capped for visual contrast.
Ant Colony Optimization (ACO) ( Gao and Schafer, 2021 )
Yes
Low ( Krishnan et al., 2023 )
Low ( Peake et al., 2019 )
Expertise-Driven
Bottleneck-Removal via Critical Path Analysis ( Bai et al., 2023 )
No
High
Low
Table 2. Comparison of representative DSE methods.
Figure 2. Lumina Framework Design
Figure 3. Examples of three task in DSE benchmark. For each task, we show the associated evidences from human system prompt, question on the left and LLM’s correct answer on the right.
Benchmark Task
Model
Accuracy (Original)
Accuracy (Enhanced)
Bottleneck Analysis
Phi-4
0.70
0.76
Qwen-3
0.73
0.80
Llama-3.1
0.47
0.53
Perf/Area Prediction
Phi-4
0.42
0.61
Qwen-3
0.59
0.82
Llama-3.1
0.23
0.39
Table 3. Accuracy across tasks and open-source LLMs
Figure 4. Mean PHV vs. Sample Efficiency among DSE Methods.
Figure 5. Distribution of PHV vs. Sample Efficiency among DSE Methods.
Figure 6. Search Pattern Comparison between ACO and Lumina
Specifications
Design A
Design B
A100
Interconnect Link Count
24
18
12
Core Count
64
96
108
Sublane Count
4
4
4
Vector Width
16
16
32
Systolic Array Height × Weight
32 × 32
32 × 32
16 × 16
SRAM Size (KB)
128
128
128
Table 4. Top-2 designs identified by Lumina Compared with NVIDIA A100