High-Performance Computing

Also known as HPC

High-performance computing studies how to make large computations run efficiently on multicore CPUs, GPUs, and distributed clusters, supporting applications from scientific simulation to large-scale AI. Common approaches include parallel programming with MPI and OpenMP, GPU kernels and compiler optimization, and algorithms that reduce data movement and communication. Scheduling, load balancing, and collective communication help coordinate work across heterogeneous machines and facilities. A central challenge is scaling performance without sacrificing correctness, portability, or efficient use of memory and network resources.

Latest papers

143 papers10 papers in the last four weeks, down 41% on the four weeks before. 0.1% of all new papers.

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†
AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
Jun 8, 2026Jaber Jaber, Osama JaberGPU Kernel Optimization High-Performance Computing
RightNow AI
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
Jun 8, 2026Sergei Vorobyov, Eugene IlyushinNeural Network Verification High-Performance Computing
Lomonosov Moscow State University · Central University
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
Jun 8, 2026Minyu Cui, Miquel PericasGPU Kernel Optimization Communication-Efficient Distributed Training
Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
Structuring agentic AI for HPC code modernization
Jun 7, 2026Anthony Marinov, Igor SfiligoiCode Translation Software Engineering Agents
San Diego Supercomputer Center University of California, San Diego La Jolla, California, United States
Twelve quick tips for designing AI-driven HPC workflows
Jun 5, 2026Jamie J. AlnasirAI for Science High-Performance Computing
Department of Computer Science Royal Holloway University of London Egham, England, UK
Attention at the Theoretical Minimum: A Mathematics of Arrays Framework for Memory-Optimal Transformer Kernels
Jun 5, 2026Lenore Mullin, Gaetan HainsMemory-Efficient Attention GPU Kernel Optimization
University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
Accelerated Fourier SAT (AFSAT): Fully Realising a GPU-based Symmetric Pseudo-Boolean SAT Solver
Jun 4, 2026Cody J Christopher, Charles GrettonGPU Acceleration High-Performance Computing
School of Computing, Australian National University, Canberra, Australia.
AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
Jun 4, 2026Xuanzhe Li, Ziyan Weng, Zhiyu Zhu +1High-Performance Computing GPU Kernel Generation
City University of Hong Kong (Dongguan) · City University of Hong Kong
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert Parallelism Expert Load Balancing
School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory +1
KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators
Jun 1, 2026Taras Sereda, Burak Bartan, Ankita Nayak +3AI Accelerator Inference GPU Kernel Optimization
Gimlet Labs Inc., San Francisco, California, USA
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
Jun 1, 2026Jie Li, Wenzhao Wu, Junqi Hu +5High-Performance Computing Code Generation
1Sun Yat-sen University · 2National Supercomputing Center in Wuxi · 3National Supercomputing Center in Shenzhen +2
Don't Let a Few Network Failures Slow the Entire AllReduce
Jun 1, 2026Peiqing Chen, Jiedong Jiang, Nengneng Yu +4Communication-Efficient Distributed Training High-Performance Computing
University of Maryland, College Park · Utrecht University · Kyoto University
On Efficient Scaling of GNNs via IO-Aware Layers Implementations
May 29, 2026Daria Fomina, Daniil Krasylnikov, Alexey Boykov +3Graph Attention Networks GPU Kernel Optimization
1Yandex · 2HSE University · 3ITMO University
HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters
May 29, 2026Yuejie Wang, Tao Chang, Yuanyuan Zhao +10Communication-Efficient Distributed Training High-Performance Computing
Peking University · Beijing Academy of Artificial Intelligence · Infrawaves +1
Caspar: CUDA Accelerator for Symbolic Programming with Adaptive Reordering
May 28, 2026Emil Martens, Aaron Miller, Matias Varnum +1Bundle Adjustment GPU Kernel Optimization
Norwegian University of Science and Technology · Skydio
FP8 is All You Need (Part 2): Full-FP64 3-D FFT on FP8-Generation Tensor CoresThe Integer-Epilogue Wall and the Minimal Hardware That Would Remove It
May 28, 2026Satoshi MatsuokaGPU Kernel Optimization GPU Acceleration
Director, RIKEN Center for Computational Science (R-CCS), Kobe, Hyogo, Japan
FP8 is All You Need (Part 1): Debunking Hardware FP64 as the HPC Holy Grail (Sep 3rd version)
May 28, 2026Satoshi MatsuokaGPU Kernel Optimization GPU Acceleration
Director, RIKEN Center for Computational Science (R-CCS) Kobe, Hyogo, Japan
KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts
May 27, 2026Shuoming Zhang, Qiuchu Yu, Ruiyuan Xu +11Agent Skill Learning GPU Acceleration
SKLP, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of New South Wales
FastKernels: Benchmarking GPU Kernel Generation in Production
May 22, 2026Gabriele Oliaro, Yichao Fu, May Jiang +5GPU Kernel Optimization GPU Acceleration
1Snowflake AI Research · 2CMU · 3UCSD +1
From Sequential Nodes to GPU Batches: Parallel Branch and Bound for Optimal $k$ -Sparse GLMs
May 21, 2026Jiachang Liu, Andrea LodiBranch and Bound GPU Kernel Optimization
∗Center for Data Science for Enterprise and Society, Cornell University, Ithaca, USA · †School of Operations Research and Information Engineering, Cornell University, Ithaca, USA · ‡Jacobs Technion-Cornell Institute, Cornell Tech and Technion–IIT, New York, USA
Real-time, EDM-inspired sonification of the activity of a supercomputer
May 21, 2026Marco Alunno, Paolo BientinesiHigh-Performance Computing Sonification
Universidad EAFIT · Carrera 49 #7sur-50 · Medelln, Colombia +3
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
May 20, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Expert Parallelism Expert Load Balancing
Instant GPU Efficiency Visibility at Fleet Scale
May 20, 2026Connor Pedersen, Dong H. Ahn, Michel Migdal +2GPU Acceleration High-Performance Computing
NVIDIA
Real-Time Parallel Counterfactual Regret Minimization
May 19, 2026Boning Li, Longbo HuangGame-Playing Agents GPU Acceleration
IIIS, Tsinghua University
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
May 19, 2026Han Guo, Jack Zhang, Arjun Menon +4GPU Kernel Optimization GPU Acceleration
1Massachusetts Institute of Technology · *Part of this work was done at Together AI. · 2Princeton University +2
A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
May 18, 2026Ruitao Liu, Xinyang Tian, Shuo Chen +4High-Performance Computing Distributed Training
Tsinghua University · Scitix AI
Lightweight Gaussian Process Inference in C++ on Metal and CUDA
May 18, 2026Yu-Hsueh FangGPU Kernel Optimization GPU Acceleration
Department of Information Management, National Taiwan University · 2H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology
Heuristic-Based Merging of HPC Traces to Extend Hardware Counter Coverage
May 15, 2026Júlia Orteu Aubach, Fabio Banchelli, Marc Clascà Ramírez +1High-Performance Computing
Barcelona Supercomputing Center (BSC), Barcelona, Spain
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM
May 15, 2026Shaoke Xi, ChonLam Lao, Boyi Jia +11High-Performance Computing Distributed Training
Alibaba Group · Harvard University · Shanghai Jiao Tong University +1

Jun 9, 2026J/K move · Enter open · S save

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Snowflake AI Research* · Seoul National University†

LLM Pruning High-Performance Computing Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Open paper AskPDF arXiv

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.