Parallel

Follow this topic Ask about this topic

Recent momentum

-26%

20 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

9 new papers

A weekly snapshot of new work published in Parallel.

Period ending 2026-09-14

5 new papers

A weekly snapshot of new work published in Parallel.

Period ending 2026-09-07

8 new papers

A weekly snapshot of new work published in Parallel.

255 papers

Latest in Parallel

Browse all topics →

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
May 12, 2026Yuanqing Wang, Yuchen Zhang, Hao Lin +9Large Models Parallel
D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
May 12, 2026Tianyu Wu, Yu Yao, Zhenting Qi +7Speculative Decoding Parallel
Efficient LLM-based Advertising via Model Compression and Parallel Verification
May 12, 2026Wenxin Dong, Chang Gao, Guanghui Yu +9Ads LLM Inference Efficiency
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1Speculative Decoding LLM Inference Efficiency
ShardTensor: Domain Parallelism for Scientific Machine Learning
May 11, 2026Corey Adams, Peter Harrington, Akshay Subramaniam +4Shard High-Performance Computing
Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration
May 11, 2026Shuzhang Zhong, Haochen Huang, Shengxuan Qiu +3Large Language Model Reasoning Tree-Of-Thoughts
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahQuadratic Attention Efficient Long-Context Inference
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Mixture-Of-Expert Inference Mixture-of-Experts Models
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
May 10, 2026Bing Xie, Zhipeng Wang, Masahiro Tanaka +1Key-Value Cache Eviction
LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces
May 9, 2026Shaun Christopher Lee, Sangeetha Abdu JyothiBackpropagation Parallel
Contextual Plackett-Luce: An Efficient Neural Model for Probabilistic Sequence Selection under Ambiguity
May 9, 2026Noam Mizrachi, Nadav Har-Tuv, Shai Shalev-ShwartzSequence Modeling Model Selection
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
May 9, 2026Benjamin L. BadgerRecurrent Model Large Models
LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection
May 9, 2026Haohui Zhang, Zhiye Wang, Xiaoying Gan +2Diffusion Language Models Parallel
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Efficiency
Parallel Lifted Planning via Semi-Naive Datalog Evaluation
May 8, 2026Dominik Drexler, Oliver Joergensen, Jendrik SeippClassical Planning Planning Methods
Regulating Branch Parallelism in LLM Serving
May 7, 2026Swapnil Gandhi, Siva Hari, William J. Dally +1Large Language Model Serving Parallel
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Mixture-Of-Expert Inference LLM Inference Efficiency
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
May 6, 2026Sajal Dash, Feiyi WangMixture-Of-Expert Inference High-Performance Computing
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Efficiency
parHSOM: A novel parallel Hierarchical Self-Organizing Map implementation
May 4, 2026Rebekah Lane, Logan Cummins, Andy Perkins +3Intrusion Detection Homomorphic Encryption
ParaRNN: An Interpretable and Parallelizable Recurrent Neural Network for Time-Dependent Data
May 4, 2026Yuxi Cai, Lan Li, Feiqing Huang +1Recurrent Neural Networks Nonparametric
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
May 4, 2026Jincheng Xie, Yawen Ling, Qi Xiao +4Large Language Model Serving Speculative Decoding
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Parallel Reinforcement Learning From Human Feedback
MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Mixture-Of-Expert Inference Prefill
VUDA: Breaking CUDA-Vulkan Isolation for Spatial Sharing of Compute and Graphics on the Same GPU
May 2, 2026Bin Xu, Pengfei Hu, Wenxin Zheng +2Cuda Rasterization
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5Large Language Model Quantization Graphics Processing Unit Memory
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
Apr 30, 2026Thea AvissEfficient Latent Reasoning Parallel
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Long-Context Inference Parallel
Efficient Training on Multiple Consumer GPUs with RoundPipe
Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2Multi-Gpu Systems Consumer Graphics Processing Units
Co-Evolving Policy Distillation
Apr 29, 2026Naibin Gu, Chenxu Yang, Qingyi Si +7Reinforcement Learning With Verifiable Reward Parallel
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTensor Programs Parallel
Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction
Apr 29, 2026Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani +2Autoregressive Decoding LLM Inference Efficiency
GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads
Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2Throughput Openclaw
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Apr 27, 2026Rezaul Karim, Austin Wen, Wang Zongzuo +3Parallel Low-Latency Systems
Multi-Robot Motions in Milliseconds: Vector-Accelerated Primitives for Sampling-Based Planning
Apr 27, 2026James D. Motes, Marco Morales, Nancy M. AmatoMulti-Robot Motion Planning Motion Planners
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Apr 26, 2026Alexis Limozin, Eduard Durech, Torsten Hoefler +2Large Language Model Reasoning Parallel
PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3Text-To-Sql Sql
ATRS: Adaptive Trajectory Re-splitting via a Shared Neural Policy for Parallel Optimization
Apr 24, 2026Jiajun Yu, Guodong Liu, Li Wang +6Trajectory Optimization Alternating Direction Method Of Multiplier
Large-Scale Data Parallelization of Product Quantization and Inverted Indexing Using Dask
Apr 23, 2026Ashley N. Abraham, Andrew Strelzoff, Haley R. Dozier +2Approximate Nearest-Neighbor Search Vector Database
FedSEA: Achieving Benefit of Parallelization in Federated Online Learning
Apr 21, 2026Harekrushna Sahu, Pratik Jawanpuria, Pranay SharmaFederated Learning Federated Reinforcement Learning
NI Sampling: Accelerating Discrete Diffusion Sampling by Token Order Optimization
Apr 20, 2026Enshu Liu, Xuefei Ning, Yu Wang +1Diffusion Language Models Diffusion Sampling
LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation
Apr 20, 2026Zepeng Sun, Naichuan Zheng, Hailun Xia +3Temporal Action Localization Temporal Deep Learning Framework
Periodic Steady-State Control of a Handkerchief-Spinning Task Using a Parallel Anti-Parallelogram Tendon-driven Wrist
Apr 20, 2026Lei Liu, Haonan Zhang, Huahang Xu +8Dynamic Rope Manipulation Periodic Domains
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Efficiency Efficient Inference
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
Apr 19, 2026Bo Li, Chuan Wu, Shaolin ZhuMixture-Of-Expert Inference Multimodal Benchmarks
A fully parallel densely connected probabilistic Ising machine with inertia for real-time applications
Apr 18, 2026Ruomin Zhu, Abhishek Kumar Singh, Jérémie Laydevant +5Two-Dimensional Ising Model Field-Programmable Gate Arrays
Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs
Apr 17, 2026Jun-Liang Lin, Kamesh Madduri, Mahmut Taylan KandemirMulti-Gpu Systems Parallel
Robust Synchronisation for Federated Learning in The Face of Correlated Device Failure
Apr 17, 2026Stefan Behfar, Richard MortierFederated Learning Fairness-Aware Learning
Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning
Apr 17, 2026Jiaxi Bi, Tongxu Luo, Wenyu Du +2Large Reasoning Models Heterogeneous Pruning Scores
Where does output diversity collapse in post-training?
Apr 17, 2026Constantinos Karouzos, Xingwei Tan, Nikolaos AletrasDiversity Large Language Model Post-Training
Breaking the Training Barrier of Billion-Parameter Universal Machine Learning Interatomic Potentials
Apr 17, 2026Yuanchang Zhou, Hongyu Wang, Yiming Du +12Interatomic Potentials High-Performance Computing
DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference
Apr 17, 2026Xiang Xia, Wuyang Zhang, Jiazheng Liu +2Diffusion Language Models Parallel
LACE: Lattice Attention for Cross-thread Exploration
Apr 16, 2026Yang Li, Zirui Zhang, Yang Liu +1Large Language Model Reasoning Parallel
A Parallel Approach to Counting Exact Covers Based on Decomposability Property
Apr 16, 2026Liangda Fang, Yaohui Luo, Delong Li +2Np-Hard Exact
DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance
Apr 16, 2026Kathiravan PalaniappanMulti-Gpu Systems Nvidia Nemotron
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Apr 1, 2026Zhensu Sun, Zhihao Lin, Zhi Chen +4Large Language Low-Latency Systems
Early-Terminable Energy-Safe Iterative Coupling for Parallel Simulation of Partitioned Port-Hamiltonian Systems
Mar 17, 2026Qi Wei, Jianfeng Tao, Hongyu Nie +1Hamiltonian Physics Lyapunov Function
ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution
Mar 3, 2026Liu Yang, Zeyu Nie, Andrew Liu +4Parallel Mock-Interface Synthesis
Randomized Kriging Believer for Parallel Bayesian Optimization with Regret Bounds
Mar 2, 2026Shuhei Sugiura, Ichiro Takeuchi, Shion TakenoBayesian Optimization Gaussian Process
Parallel Reference-Centric Continuous-Time Relative Localization with Augmented Clamped Non-Uniform B-Splines
Feb 25, 2026Jiadong Lu, Zhehan Li, Tao Han +4Simultaneous Localization And Mapping B-Spline

May 12, 2026 · cs.LGJ/K move · Enter open · S save

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training

Yuanqing Wang, Yuchen Zhang, Hao Lin, Junhao Hu+8

Peking University · Infinigence AI · Institute of Computing Technology, CAS · Shanghai Jiao Tong University · Tsinghua University

Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness.

Large Models Parallel Trainability Reconfigurable

Open paper Ask about it PDF