Conformer

Ask about this topic

Latest papers

CardsList

12 papers3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Oct 9, 2026

From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

Broadcast archives offer scale and diversity for self-supervised learning (SSL) but contain substantial noisy, non-speech, and multilingual content. Prior work typically relies on voice activity detection or language identification (LID) to extract speech, ensuring presence rather than quality. Whether and how to filter unlabeled speech by quality for SSL, particularly from noisy broadcasts, remains underexplored. We present a large-scale Finnish SSL study starting from 2.1M hours of unconstrained broadcast audio. We propose automatic speech recognition (ASR) confidence as a proxy for pretraining data quality and show 3-7% relative WER improvements over LID-based filtering despite a smaller training set. We further address the lack of controlled architectural comparisons in speech SSL by evaluating Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets. Our models improve upon prior Finnish SSL baselines, particularly at Base scale and out-of-domain.

Yaroslav Getman, Tamás Grósz, Tommi Lehtonen +1

Dept. lnformation and Communications Engineering Aalto University Espoo, Finland · Walton Institute, South East Technological University, Ireland Aalto University Espoo, Finland · Finnish National Audiovisual Institute (KAVI) Helsinki, Finland

ConformerSpeech ProcessingSelf-Supervised Speech Representation LearningTransformerAutomatic Speech RecognitionLow-Resource Speech Recognition
AskPDF
Sep 27, 2026

Deep Learning Techniques for Phoneme Recognition in Italian Children' s Speech

Speech therapists often face difficulties diagnosing impairments due to the lack of efficient tools for transcribing speech into the International Phonetic Alphabet (IPA). This work addresses this challenge with Broca, a Conformer-based deep learning system pretrained on 8 days of adult speech and fine-tuned on a 165-minute dataset of Italian child speech collected through a range of standardized diagnostic tests for children aged 3.5-6.5. Broca was optimized to handle phonetic variability in children's speech, including tone, accent, and speech errors, and achieved a state-of-the-art weighted Phoneme Error Rate of 13.36% on Italian speech. Remarkably, this performance was obtained using less than three hours of child-specific data, underscoring the model's efficiency and robustness in low-resource clinical settings. This work demonstrates that accurate, vocabulary-independent speech-to-IPA transcription can be achieved with minimal data, paving the way for more accessible, data-efficient tools to support speech assessment and diagnosis.

Nicola Barbaro, Cristina Gena, Francesco Petriglia +3

University of Turin, Computer Science Department, Via Pessinetto 12, 10149 Turin, Italy · Fondazione Paideia Ente Filantropico, Via Moncalvo 1, 10131 Turin, Italy

ConformerAutomatic Speech RecognitionClinical Speech ProcessingLow-Resource Speech Recognition
AskPDF
Sep 17, 2026

Design of the IBM Granite 5.0 TurboCTC ASR Model

We describe the architecture, training methodology and inference speedups of Granite 5.0 Turbo CTC, a 470 million parameter encoder-only model with an excellent speed-accuracy tradeoff. The architecture uses pyramidal temporal subsampling within Conformer blocks using strided depthwise convolutions, block-diagonal (chunk-wise) self-attention, and conditioning on intermediate predictions from the middle layer. Training highlights are the use of only publicly available data, the novel use of a Muon optimizer, and balanced data sampling. Inference speedups include replacing 1 x 1 convolutions with linear layers and optimizing the attention computation in the Conformer blocks. Collectively, these result in a model that is on the speed-accuracy Pareto frontier of the Open ASR leaderboard for English short-form ASR while being twice as fast as the fastest competitor. The model can be used under a permissive license and downloaded from https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc.

Brian Kingsbury, George Saon, Masayuki Suzuki +5

IBM Research

TransformerConnectionist Temporal ClassificationAutomatic Speech RecognitionConformer
AskPDF
Sep 3, 2026

The Anatomy of an ASR Hallucination

ASR systems sometimes produce fluent text that is unrelated to the speech they receive. We view these hallucinations as one possible consequence of a broader grounding failure, in which the transcript is no longer adequately guided by the audio. To understand where this failure becomes possible, we study two independently trained Conformer-Large recognizers - one CTC and one RNN-T - under environmental degradation and speaker-background shift. In both models, the final encoder stage emerges as a critical boundary: bypassing the final block causes divergence on nearly every utterance, whereas bypassing middle blocks has little effect. At this same stage, the representations become more compact, text becomes readable by the trained decoder, and grapheme information becomes explicit. Importantly, the intervention produces garbled or repetitive output rather than fluent fabrication. Our result therefore identifies a mechanistic precondition for hallucination - the failure to produce adequately grounded output - not the complete origin of naturally occurring hallucinations. Together, the results reveal a consistent terminal-stage dependency for grounded recognition across two decoder families and multiple distribution shifts.

Hamees Sayed, Apoorv Singh, Kumar Aman +1

Smallest AI · Fast Code AI

Robust ASRConformerAutomatic Speech RecognitionMechanistic Interpretability
AskPDF
Jul 23, 2026

EnsembleEGNN: Set-Based Graph Learning for Thermodynamic Ensembles of Cyclic Peptides

Molecular graph encoding often relies on a single, static structure, ignoring the thermodynamic ensemble of molecules that are present in solution. Here, we introduce EnsembleEGNN, a foundation model that encodes structural ensembles by processing individual conformers through shared equivariant graph neural network layers, pooled with a set attention block, to make property predictions from the whole ensemble. Pretrained on the CREMP cyclic peptide dataset using multi-task self-supervision, the model is trained to encode the conformational variability of each molecule. When predicting membrane permeability from the CycPeptMPDB benchmark, EnsembleEGNN achieves an R2R^2 of 0.4770.477 under random cross-validation, outperforming a sequence-only BERT baseline (R2=0.439R^2=0.439). This representation advantage persists under rigorous out-of-distribution Butina splits (R2=0.401R^2=0.401 versus 0.3540.354). Finally, a hybrid architecture co-training EnsembleEGNN with the BERT model achieves the highest overall accuracy across both random (R2=0.538R^2=0.538) and structural holdout evaluations (R2=0.444R^2=0.444). These results demonstrate that encoding conformational ensembles into latent representations improves predictions for properties governed by thermodynamics.

Aaron L. Feller, Kris Deibler, Maxim Secor

Interdisciplinary Life Sciences, University of Texas at Austin, Austin, TX. · Molecular AI, Novo Nordisk, Lexington, MA.

Equivariant GNNsMolecular Property PredictionMolecular Representation LearningConformer
AskPDF
Jul 21, 2026

Boltzmann-Expected Molecular Design with Decoupled Annealing Flows

Most 3D properties relevant to molecular design, including free energies and shape descriptors, are expectations\textit{expectations} over the Boltzmann distribution over 3D configurations of a molecular graph. However, existing property-guided generative models tie each property to a single structure, ignoring the underlying ensemble. We recast 3D molecular design as Boltzmann-expected design\textbf{Boltzmann-expected design} and realise it with DECAF\textbf{DECAF} (Decoupled Annealing Flows), which factorise the joint distribution over graphs and coordinates into two conditional flow models: a graph-conditioned flow p(x∣G)p(x\mid\mathcal{G}), acting as a Boltzmann emulator\textit{Boltzmann emulator}, and a coordinate-conditioned flow p(G∣x)p(\mathcal{G}\mid x), proposing new graphs from 3D information. By alternating the two flows, DECAF optimises molecular graphs with a simulated-annealing acceptance rule whose scoring function is evaluated on ensembles drawn from p(x∣G)p(x\mid\mathcal{G}), making ensemble statistics, not single-conformer properties, the design target. The resulting loop requires no retraining to change objectives. On GEOM-Drugs, we show that ensemble-aware optimisation produces graphs whose mean radius of gyration and solvent-accessible surface area consistently shift toward targets, while single-conformer optimisation degrades on larger drug-like molecules where Boltzmann distributions are broadest. DECAF extends to multi-objective trade-offs and, uniquely among 3D generative models, to higher-moment design\textbf{higher-moment design}: jointly optimising an ensemble property's variance and skewness to produce flexible molecules biased to a prescribed conformational regime: we verify the conformational distributions of these higher-moment designs with all-atom MD simulations.

Selma Moqvist, Richard Beckmann, Ross Irwin +2

Department of Computer Science and Engineering Chalmers University of Technology & University of Gothenburg SE-41296 Gothenburg, Sweden · Molecular AI Discovery Sciences, R&D AstraZeneca SE-43183 Mölndal, Sweden

Molecular OptimizationMolecular Graph Generation3D Molecular GenerationMolecular GenerationMolecule DesignConformer
AskPDF
Jul 11, 2026

Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimization

Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.

Shuhai Peng, Jinjiang Liu, Hui Lu +5

Tsinghua University · The Chinese University of Hong Kong · SenseTime

Target Speaker ExtractionSpeech ProcessingConformer
AskPDF
Jun 17, 2026

QC-GAN: A Parameter-Efficient Quaternion Conformer GAN for High-Fidelity Speech Enhancement

We propose a parameter-efficient speech enhancement framework, Quaternion Conformer GAN (QC-GAN), which combines a Quaternion Conformer generator with MetricGAN-based training. The Hamilton product encodes the magnitude and phase via structured weight sharing, reducing the number of layer parameters while preserving their interdependencies. A metric-learning discriminator was employed to maximize perceptual quality by optimizing the approximate perceptual evaluation scores. On the VoiceBank+DEMAND dataset, QC-GAN achieved a Perceptual Evaluation of Speech Quality (PESQ) score of 3.48 with only 0.89M parameters, delivering a performance comparable to state-of-the-art models at less than half their size. A 35K-parameter variant achieved a PESQ score of 3.23, surpassing conventional methods with significantly fewer parameters. Evaluation on the DNS-Challenge 3 dataset further confirmed generalization to real-world conditions.

Shogo Yamauchi, Hideaki Tamori, Makoto Sakai +2

The Asahi Shimbun Company, Japan · Tokyo Woman’s Christian University, Japan

TransformerComplex-Valued Neural NetworksSpeech EnhancementGenerative Adversarial NetworksConformer
AskPDF
May 23, 2026

End-to-End Intracortical Speech Decoding from Neural Activity

Current high-performing intracortical speech neuroprostheses achieve low word error rates but typically rely on external language models during inference, increasing memory, computation, and latency. In this work, we investigate whether meaningful character-level decoding is achievable without such models. We propose an end-to-end Conformer-based neural decoder trained directly on intracortical recordings from a participant with amyotrophic lateral sclerosis (ALS). Without any external language model, the system achieves a character error rate (CER) of 23.80% on held-out validation data. Analysis shows that performance variability is driven by inter-session signal degradation, while dominant errors arise from incorrect word boundary segmentation. These results demonstrate that effective character-level decoding is possible in a fully end-to-end framework, providing a strong neural signal for downstream linguistic processing.

Owais Mujtaba Khanday, Jose A. Gonzalez-Lopez, Marc Ouellet +2

1Dpt. of Signal Theory, Telematics and Communications, University of Granada, Spain · 2Brain, Mind, and Behavior Research Center, University of Granada, Spain · 3Grupo CSUR de Epilepsia Refractaria, Hospital Virgen de Las Nieves (Granada), Spain

Brain-Computer InterfacesNeural DecodingAutomatic Speech RecognitionConformerBrain-to-Speech Decoding
AskPDF
May 7, 2026

FlashMol: High-Quality Molecule Generation in as Few as Four Steps

Generating chemically valid 3D molecular conformations is critical for computational drug discovery. Classical diffusion-based models like GeoLDM perform well but require hundreds of steps, making large-scale in silico screening impractical. Recent efforts on few-step molecular generation have accelerated this process to 12-50 steps, but they often largely sacrifice sample stability. In this work, we present FlashMol, an ultra-fast molecule generative model producing high-quality molecular conformations in as few as 4 steps. To achieve this, we adapt distribution matching distillation (DMD) - a reverse KL-divergence minimization objective - to the molecular domain for effective distillation. Considering the local minimization behavior of DMD, we respace the molecule generation timesteps, providing the generator with much better initialization and enables effective distillation. Additionally, to mitigate the mode-seeking behavior of DMD and improve diversity, we further regularize it with a Jensen-Shannon divergence term, which incorporates the mean-seeking behavior of the forward KL divergence. Extensive experiments on QM9 and GEOM-DRUG datasets demonstrate that FlashMol matches and even surpasses the original 1000-step teacher, achieving up to 250×\times acceleration in sampling speed while maintaining high molecular quality.

Xinyuan Wei, Zian Li, Shaoheng Yan +2

Institute for Artificial Intelligence, Peking University · Yuanpei College, Peking University · School of Intelligence Science and Technology, Peking University +2

Diffusion DistillationFew-Step Diffusion Sampling3D Molecular GenerationMolecular GenerationDistribution Matching DistillationConformer
AskPDF
Apr 7, 2026

iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding

Brain-computer interfaces (BCIs) for speech restoration hold transformative potential for the approximately 173,000--232,500 individuals worldwide with ALS-related dysarthria. Despite recent progress, high-performance speech BCIs have been demonstrated in only 22--31 patients globally, largely due to limitations in neural decoding accuracy and practical input interfaces. We present iPhoneme, a brain-to-text communication system that jointly addresses these challenges through integrated modeling and interaction design. The system combines a deep learning phoneme decoder based on a modified Conformer architecture (ConformerXL, 192.9M parameters) with a gaze-assisted phoneme input interface that mitigates the Midas touch problem in eye-tracking systems. The acoustic model incorporates a temporal prenet with multi-scale dilated convolutions and bidirectional GRU for neural jitter correction, temporal subsampling for CTC stability, and Pre-RMSNorm stabilization across 12 encoder blocks, trained with AdamW and cosine scheduling. On the interaction side, iPhoneme introduces a chorded gaze-plus-silent-speech paradigm that replaces dwell-time selection, enabling more efficient input. We evaluate the system on the T15 dataset (45 sessions, 8,071 trials) of 256-channel intracranial EEG from speech motor cortex regions. A 6-gram phoneme language model trained on 3.1M sequences, combined with WFST beam search (beam=128), achieves 92.14% phoneme accuracy (7.86% PER) and 73.39% word accuracy (26.61% WER), approximately 3% above prior state-of-the-art. The system operates on CPU with 180 ms latency, demonstrating real-time, high-accuracy brain-to-text communication for ALS.

Yoonmin Cha, Dawit Chun, Sung Park

School of Data Science and Artificial Intelligence, Taejae University

ConformerBrain-Computer InterfacesNeural DecodingTransformer
AskPDF
Mar 22, 2026

CIPHER: Conformer-based Inference of Phonemes from High-density EEG

Decoding speech information from scalp EEG remains difficult due to low SNR and spatial blurring. We present CIPHER (Conformer-based Inference of Phonemes from High-density EEG Representations), a dual-pathway model using (i) ERP features and (ii) broadband DDA coefficients. On OpenNeuro ds006104 (24 participants, two studies with concurrent TMS), binary articulatory tasks reach near-ceiling performance but are highly confound-vulnerable (acoustic onset separability and TMS-target blocking). On the primary 11-class CVC phoneme task under full Study 2 LOSO (16 held-out subjects), performance is substantially lower (real-word WER: ERP 0.671 +/- 0.080, DDA 0.688 +/- 0.096, indicating limited fine-grained discriminability. We therefore position this work as a benchmark and feature-comparison study rather than an EEG-to-text system, and we constrain neural-representation claims to confound-controlled evidence.

Varshith Madishetty

EEG-to-Text DecodingConformerEEG DecodingCross-Subject EEG Decoding
AskPDF