Gumbel-Softmax Relaxation

Ask about this topic

Momentum

12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 58

All topics
CardsList

  1. Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersion

    Oct 7, 2026Walid Bendada, Guillaume Salha-GalvanGumbel-Softmax RelaxationOptimal Sample Complexity

    Spotify · SJTU Paris Elite Institute of Technology

  2. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Gumbel-Softmax RelaxationPretraining

    National University of Singapore · Princeton University · Nanyang Technological University

  3. Underscoring the Problem: Why Softpick Fails at Initialization

    Oct 4, 2026Aryan Sood, Jaikaran Singh, Ishaan BansalGumbel-Softmax Relaxation

    IIT Roorkee

  4. Does a Shared Temperature Imply a Shared Angular Scale in Probabilistic Contrastive Learning?

    Sep 30, 2026Ningkang Peng, Qianfeng Yu, Jingyang Mao +4Contrastive LearningCosine Similarity

    Nanjing Normal University · Nanjing University of Chinese Medicine · Tohoku University

  5. When a Flatness Proxy Is Not a Function: Robustness Certificates and Training Interventions

    Sep 29, 2026Vicente Opazo, Jose Calatayud-Mateu, Cristobal Rojas +1Gumbel-Softmax Relaxation

    Centro Nacional de Inteligencia Artificial (CENIA) · Barcelona Supercomputing Center (BSC) · Pontificia Universidad Cat´olica de Chile

  6. Dual-Stream Simultaneous Translation via 2D Grid Attention

    Sep 28, 2026Yu Pu, Wei-Qiang ZhangBidirectionalFast Inference

    Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China

  7. Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks

    Sep 28, 2026Manya Singh, Arjun PakrashiAmbiguityMulti-Label Classification

    School of Computer Science, University College Dublin, Ireland

  8. Pretraining Transformers with Quantized Softmax in Attention

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiTransformer AttentionGumbel-Softmax Relaxation

    University of Illinois Urbana-Champaign

  9. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiGumbel-Softmax RelaxationTransformer Architectures

    University of Illinois Urbana-Champaign

  10. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

    Sep 27, 2026Sriman AchantaBlock Sparse Flash AttentionAutoregressive Decoding

    Virginia Commonwealth University

  11. FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

    Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuBlock Sparse Flash AttentionGumbel-Softmax Relaxation

  12. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangGumbel-Softmax RelaxationGating

    St. John Fisher University, Rochester, New York, USA.

  13. Mixed-Integer Nonlinear Differentiable Predictive Control for Underground Pumped Hydro Energy Storage Systems

    Sep 16, 2026Honghui Zheng, Ján Boldocký, Yury Dvorkin +1Model Predictive ControlNeural Policies

    Johns Hopkins University, Baltimore MD, US · Slovak University of Technology in Bratislava, Slovakia

  14. EFQ-Softmax: Exp-Free Quantization for Softmax

    Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Gumbel-Softmax RelaxationBlock Sparse Flash Attention

    Xi'an Jiaotong University · Xi’an Jiaotong University · Huawei Technologies Co., Ltd +1

  15. Soft-Argmax for the Projective Plane via the Veronese Embedding

    Sep 1, 2026Benjamin El-Zein, Dominik Eckert, Paul Zech +4ProjectionSpherical Latent Space

    Siemens Healthineers AG, X-ray Products, Forchheim, Germany · Artificial Intelligence Lab, Otto-von-Guericke-University, Magdeburg, Germany

  16. Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

    Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangLow-Rank StructureSpectral Norm

    Nanyang Technological University, Singapore · Carnegie Mellon University

  17. SoftWater: Class-Aware Rate Allocation for Softmax Quantization

    Aug 12, 2026Joao V. Cavalcanti, Ashia C. WilsonGumbel-Softmax RelaxationKullback-Leibler Divergence

    Department of Electrical Engineering and Computer Science · Massachusetts Institute of Technology · Cambridge, MA 02139, USA

  18. Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention

    Aug 11, 2026Vicente OpazoFeature LearningLow-Rank Structure

    CENIA

  19. A Quantum Roadmap for Softmax Attention: Exact Born-Rule Analogs for Softmax Attention on the Probability Simplex

    Aug 11, 2026Eric A. F. Reinhardt, Adam J. HauserGumbel-Softmax RelaxationQubit

    Department of Physics and Astronomy University of Alabama, Tuscaloosa, AL 35487, USA

  20. Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training

    Aug 11, 2026Artyom Sabitov, Daniil Volkov, Alexey ZaytsevReal-World Content Recommendation ProblemBounded-Memory

    Applied AI Institute, Moscow, Russia Risk department

  21. Fast LapSum: Exact Differentiable Top-kk at Million Scale

    Aug 7, 2026Jakub Antczak, Joanna Wojciechowicz, Kamil Książek +3Top-KDecode Speedup

    Wrocław University of Science and Technology, Wrocław, Poland. · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland.

  22. Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers

    Aug 5, 2026Zhen Zhang, Amr AlanwarMulti-Head AttentionTransformer Architectures

    School of Computation, Information and Technology Technical University of Munich, Germany

  23. Stochastic Sequential Search in Very-High-Dimensional Feature Selection

    Aug 2, 2026Petr Somol, Jiří GrimFeature SelectionHigh-Dimensional

    Institute of Information Theory and Automation, The Czech Academy of Sciences Prague, Czech Republic

  24. Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages

    Aug 2, 2026Saierdaer Yusuyin, Nanling Jiang, Hao Huang +1Multilingual Automatic Speech RecognitionGrapheme-To-Phoneme

    School of Computer Science and Technology, Xinjiang University, China · Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, China +1

  25. Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

    Jul 26, 2026Mihai Suteu, Ovidiu SerbanEnsembleGumbel-Softmax Relaxation

    Data Science Institute, Imperial College London

  26. The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

    Jul 25, 2026Byeong Hoon YoonTransformer AttentionTransformer Architectures

    Independent Researcher

  27. Generalised Balanced Softmax: A Finite-Data Perspective on Logit Adjustment for Long-Tailed Recognition

    Jul 24, 2026Yi-Hang Zhu, Rajeev Raman, Shiqi Su +4Imbalanced ClassificationGumbel-Softmax Relaxation

    School of Computing and Mathematical Sciences, University of Leicester, Leicester, LE1 7RH, UK · Department of Physics and Astronomy, University of Leicester, Leicester, LE1 7RH, UK · Scientific Computing, Rutherford Appleton Laboratory, Science and Technology Facilities Council, Didcot, OX11 0QX, UK +2

  28. Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions

    Jul 15, 2026Wisdom DogahGumbel-Softmax Relaxation

    Faculty of Computing and Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana. · BlackMatrix AI Research, Accra, Ghana.

  29. A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition

    Jul 10, 2026Juan Terven, Diana Margarita Córdova Esparza, Julio Alejandro Romero Gonzalez +4Few-Shot LearningImagenet

    CICATA Querétaro Instituto Politecnico Nacional Mexico · Facultad de Informática Universidad Autónoma de Querétaro Mexico

  30. Sensitivity to Subjective Expected Utility Maximization: A Methodological Study, with an Illustrative Application to LLM Decision-Making

    Jul 8, 2026Jeff HelznerUtility MaximizationUncertainty Score

  31. Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detection

    Jul 6, 2026Víctor Yeste, Paolo RossoMulti-Label ClassificationIntermediate Decoder Layers

    1PRHLT Research Center, Universitat Politècnica de València, Spain · School of Science, Engineering and Design, Universidad Europea de Valencia, Spain · 3Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI)

  32. Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

    Jul 3, 2026Hamish OgilvyLarge Language Model QuantizationAutoregressive Decoding

    Algolia Research

  33. Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domains

    Jun 9, 2026Zilai Wang, Natarajan Balaji Shankar, Mohan Shi +2Speech EncoderDomain Adaptation

    University of California, Los Angeles, USA

  34. Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

    May 29, 2026Jabin Koo, Hoyoung Kim, Minwoo Jang +1Preference LearningPreference Alignment Learning

    Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea

  35. A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification

    May 21, 2026Marcel Kühn, Yoon Thelge, Bernd RosenowBatchGumbel-Softmax Relaxation

    Institute for Theoretical Physics, Leipzig University · 2ScaDS.AI Dresden/Leipzig

  36. Reading Calibrated Uncertainty from Language Model Trajectories

    May 19, 2026Aliai Eusebi, Alexander Herzog, Xiaoyu Liang +3Large Language Model UncertaintyCalibrated Uncertainty

    University College London, London, United Kingdom · Core46, London, United Kingdom

  37. The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

    May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer ArchitecturesGumbel-Softmax Relaxation

    Department of Mathematics, University of Tübingen, Germany.

  38. A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention

    May 12, 2026Tomohiro Hayase, Ryo KarakidaGumbel-Softmax RelaxationLong-Tailed Distribution

    AIST

  39. The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods

    May 10, 2026Sanket Badhe, Priyanka Tiwari, Deep ShahGumbel-Softmax RelaxationOverconfidence

    Google

  40. Fitting Multilinear Polynomials for Logic Gate Networks

    May 9, 2026Youngsung KimDifferentiable Logic Gate NetworksPolynomials

    Department of AI; and Department of ECE, Inha University

  41. Scaling Limits of Long-Context Transformers

    May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Linear AttentionEfficient Long-Context Inference

  42. Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization

    May 8, 2026Navid Rezazadeh, Arash Gholami DavoodiGumbel-Softmax RelaxationTransformer Attention

    University of California, Irvine · Carnegie Mellon University

  43. Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning

    May 8, 2026Zixuan Xie, Xinyu Liu, Claire Chen +3In-Context LearningLinear Attention

    University of Virginia · California Institute of Technology · Purdue University

  44. Streaming Adversarial Robustness in Fuzzy ARTMAP: Mechanism-Aligned Evaluation, Progressive Training, and Interpretable Diagnostics

    May 7, 2026Shane Cairns, Leonardo Enzo Brito da Silva, Sasha Petrenko +2Adversarial TrainingGumbel-Softmax Relaxation

    Department of Electrical and Computer Engineering, Missouri University of Science and Technology, Rolla, MO, USA · Instituto Metrópole Digital, Universidade Federal do Rio Grande do Norte, Natal, RN 59078-900, Brazil · Kummer Institute Center for Artificial Intelligence and Autonomous Systems (KICAIAS), Missouri University of Science and Technology, Rolla, MO, USA

  45. Linearizing Vision Transformer with Test-Time Training

    May 4, 2026Yining Li, Dongchen Han, Zeyu Liu +3Self-Supervised Vision TransformersVision Transformer

    Tsinghua University, Beijing, China

  46. Soft-to-Hard Routing in Sparse Mixture-of-Experts Models

    May 4, 2026Reza RastegarGumbel-Softmax RelaxationApproximate Nearest-Neighbor Search

    Meta Platforms, Inc Woodinville, WA

  47. Stochastic Sparse Attention for Memory-Bound Inference

    May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5Dynamic Sparse AttentionAutoregressive Decoding

    Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA

  48. Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models

    Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Wireless Foundation ModelsGumbel-Softmax Relaxation

    Merck & Co., Inc., Cambridge, MA, USA

  49. Transformer Approximations from ReLUs

    Apr 27, 2026Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1Gumbel-Softmax RelaxationRectified Linear Unit

    Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1

  50. ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers

    Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Kimi Delta AttentionGumbel-Softmax Relaxation

    Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University

  51. Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices

    Apr 26, 2026Dawon Choi, Hana Kim, Ji-Hoon KimBatch NormalizationGumbel-Softmax Relaxation

    Dept. of Artificial Intelligence Semiconductor Engineering, Hanyang University, Seoul, Republic of Korea · Dept. of Electronic Engineering, Hanyang University, Seoul, Republic of Korea

  52. Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding

    Apr 23, 2026Jon-Paul CacioliCross-Entropy LossesGumbel-Softmax Relaxation

    Independent Researcher, Melbourne, Australia

  53. GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

    Apr 20, 2026Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4Large Language Model QuantizationPost-Training Quantization

    ISTA · ETH Zürich · ISTA & Red Hat AI +1

  54. TabICLv2: A better, faster, scalable, and open tabular foundation model

    Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Tabular Foundation ModelsPretraining

    SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France

  55. Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators

    Feb 9, 2026Gunn KimEntropyGumbel-Softmax Relaxation

    Department of Physics, Sejong University, Seoul 05006, Republic of Korea

  56. FlexAct: Why Learn when you can Pick?

    Jan 10, 2026Ramnath Kumar, Kyle Ritscher, Junmin Judy +2Activation FunctionsGumbel-Softmax Relaxation

    University of California Los Angeles. · University of California Los Angeles, Arena.

  57. The EM-algorithm and the Method of Moments in Softmax Mixture Models

    Sep 16, 2024Xin Bing, Florentina Bunea, Jonathan Niles-Weed +1Mixture ModelsExpectation-Maximization

    Department of Statistical Sciences, University of Toronto · Department of Statistics and Data Science, Cornell University · Courant Institute of Mathematical Sciences and Center for Data Science, New York University +1

  58. Smoothing the Score Function to Enhance Generalization in Diffusion Models

    Date pendingXinyu Zhou, Jiawei Zhang, Stephen J. WrightScore-Based Diffusion ModelGumbel-Softmax Relaxation

    Department of Computer Sciences, University of Wisconsin Madison, WI, USA.