Soft-Token Representations

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 27

All topics

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs
Sep 29, 2026Runqiu Xu, Zhisheng Zheng, David HarwathVoice Agents Speaker
The University of Texas at Austin, Austin, TX, USA
Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis
Sep 29, 2026Yulin Sun, Kele Xu, Yong DouMultimodal Sentiment Analysis Multimodal Fusion
National University of Defense Technology
CASS: Contribution-Aware Structured Sparsity for Model Merging
Sep 28, 2026Yan Li, Guiping Cao, Meng Xu +5Continual Model Merging Soft-Token Representations
Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
SMAT: Simple and Efficient Merge-Aware Training
Sep 27, 2026Yanggan Gu, Yuanyi Wang, Zhen Li +5Continual Model Merging Soft-Token Representations
The Hong Kong Polytechnic University (PolyU) · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1
MeRoTune: RoPE-Safe Merging with a Tunable Dial
Sep 7, 2026Salman FarozRotary Position Embedding Soft-Token Representations
Can LLMs Use Relational Transformer Embeddings?
Aug 31, 2026Francisco Galuppo Azevedo, Clarissa Lima LouresSoft-Token Representations
Kunumi Institute, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
MergeOver: Post-Training Token Merging for Recursive Vision Transformers
Aug 13, 2026Junseo Kim, Uraz Odyurt, Amirreza YousefzadehSelf-Supervised Vision Transformers Vision Transformer
Computer Architecture for Embedded Systems, University of Twente Enschede, The Netherlands
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Token Compression Video-Language Models
1Evermind · 2Shanda Group · 3Peking University
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis
Jul 17, 2026Zhenqi Jia, Yuan Zhao, Aruukhan +2Emotion Recognition Soft-Token Representations
Inner Mongolia University · The Chinese University of Hong Kong
Tokenizing Numerical and Embedding Features for LLM RecSys
Jul 10, 2026Zhe Xu, Ankit Peshin, Chiyu Zhang +7Real-World Content Recommendation Problem Soft-Token Representations
Meta
Soft Token Alignment for Cross-Lingual Reasoning
Jun 25, 2026Jiayi He, Jungsoo Park, Wei Xu +1Multilingual Language Models Soft-Token Representations
Georgia Institute of Technology
Token-Operations-Oriented Inference Optimization Techniques for Large Models
Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM Inference Optimization Soft-Token Representations
Unicom Data Intelligence, China Unicom · Data Science and Artificial Intelligence Research Institute, China Unicom · Tsinghua University +7
An Integrable Token Mixing Layer from the Generalized Yang Baxter Equation
Jun 13, 2026Snigdha Chandan KhilarAsymmetric Grouped-Attention Variate Mixer Efficient Sequence Mixers
Independent Researcher
Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning
Jun 10, 2026Zhirui Chen, Ziwei Chen, Ling ShaoMultimodal Large Language Models In-Context Learning
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · ByteDance Inc.
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
Jun 6, 2026Kyumin Choi, Ikbeom JangVisual Token Pruning Self-Supervised Vision Transformers
Hankuk University of Foreign Studies Seoul, South Korea
Leveraging Soft Distributions of SSL-Derived Discrete Speech Tokens for Downstream Inference
Jun 5, 2026Kentaro Onda, Satoru Fukayama, Daisuke Saito +1Soft-Token Representations Automatic Speech Recognition
The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan
Saliency-Aware Model Merging
May 30, 2026Jungin Park, Jiyoung Lee, Kwanghoon SohnContinual Model Merging Multi--Task Learning
Yonsei University, Seoul, South Korea · Ewha Womans University, Seoul, South Korea.
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
May 28, 2026Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji +2Diffusion Language Models LLM Inference Optimization
Georgia Institute of Technology
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiSoft-Token Representations Vision Transformer
Electronic Engineering · Soongsil University · Seoul, South Korea
Action-Prior Denoising for Smooth Real-Time Chunking
May 25, 2026Dongyang Liu, Zhaowen Zheng, Yu Sun +3Action Chunks Chunk
ROKAE (Shandong) Robot Group Co., Ltd. · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
May 23, 2026Juan Ignacio Bustos Gorostegui, Maria Elena BuemiOpen-Vocabulary Action Recognition 3D Hand
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC). · CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC). Argentina.
DORA: Dynamic Online Reinforcement Agent for Token Merging in Vision Transformers
May 12, 2026Kaixuan He, Song Chen, Yi KangVision Transformer Soft-Token Representations
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Decoupled DiLoCo for Resilient Distributed Pre-training
Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Distributed Optimization Cpu-Gpu Hybrid Designs
1Google DeepMind · 2Google Research
Compressing Sequences in the Latent Embedding Space: $K$ -Token Merging for Large Language Models
Apr 16, 2026Zihao Xu, John Harvill, Ziwei Fan +3Large Language Model Compression Task-Aware Compression
♢Rutgers University · ♡Mistral AI · ♠AWS AI Labs +3
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Soft-Token Representations Segment Anything Model
University of Stuttgart · DFKI · Max Planck Research School +10
AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Feb 18, 2026David Smerkous, Zian Wang, Behzad Najafian3D Masked Autoencoders Self-Supervised Vision Transformers
Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA
FuseMamba-VD: Dual Branch VideoMamba with Gated Class Token Fusion for Violence Detection
May 23, 2025Damith Chamalke Senadeera, Muhammad Awais, Shibo Li +2Fine-Grained Video Understanding Violence Detection
Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK · Queen’s University Belfast, UK

Sep 29, 2026, eess.ASJ/K move · Enter open · S save

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

Runqiu Xu, Zhisheng Zheng, David Harwath

The University of Texas at Austin, Austin, TX, USA

Voice Agents Speaker Frozen Language Model Strong Large Language Model Soft-Token Representations

Open paper AskPDF arXiv

Multi-user voice agents must track who said what across dialogue sessions. Text LLMs are attractive backbones for such agents, but transcripts alone do not expose acoustic speaker identity, leaving the model without a persistent reference for linking information to speakers across sessions. We address this gap by introducing Speaker Handles, soft-token representations that expose acoustic speaker identity to a frozen text LLM for cross-session speaker-dependent reasoning. A three-stage curriculum trains a lightweight projector, with fewer than 0.1% of the backbone's parameters, to map speaker embeddings into these handles. Establishing whether the resulting handles truly support cross-session speaker-dependent reasoning is challenging with existing benchmarks because textual cues can partially reveal fact ownership. We therefore present SpeakerBind, a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution. Speaker Handles achieve 97.40-98.36% accuracy on VoxCeleb1 and 70.40% on SpeakerBind, close to the 71.88% topline. These results show that the proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning.

Momentum

Related topics

Latest papers 27