OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs
Abstract
Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.
Figures & tables
| Task | Model | Public Test Set | Metric | Public Result |
|---|---|---|---|---|
| ASR | Qwen2-Audio | WenetSpeech(n/m) AISHELL-2(m/i/a) LibriSpeech (c/o) | WER/CER (%) | 8.84 / 8.40 3.0 / 3.0 / 2.9 1.6 / 3.6 |
| OSUM | 6.46 / 5.34 2.81 / 2.75 / 2.73 2.19 / 5.53 | |||
| OSUM-Pangu | 7.40 / 10.49 3.01 / 2.98 / 2.95 3.51 / 8.36 | |||
| VED | Qwen2-Audio | VocalSound | ACC (%) | 93.3 |
| OSUM | 82.58 | |||
| OSUM-Pangu | 73.04 |
| Model | IFR (%) |
|---|---|
| Qwen2Audio-Instruct | 71.3 |
| OSUM-Pangu (Ours) | 90.2 |
| Task | Test | FI | NL | |
|---|---|---|---|---|
| ASR | test-net/librispeech-clean | 7.36/3.64 | 7.40/3.51 | +0.04/-0.13 |
| SER | Test emotion | 67.39 | 67.41 | +0.02 |
| SGC | Test gender | 97.04 | 96.02 | -1.02 |
| SRWT | Test align | 22.39 | 17.52 | -4.87 |
| SSR | Test style | 62.79 | 58.05 | -4.74 |
| VED | Test event | 77.74 | 73.04 | -4.70 |
| Model | LLaMA Q | TriviaQA | Web Q |
|---|---|---|---|
| ChatGPT-4o | 71.7 | 69.7 | 51.6 |
| GLM-4-Voice | 50.7 | 26.5 | 15.9 |
| DeepTalk | 59.7 | 27.5 | 23.1 |
| OSUM-EChat | 55.3 | 33.7 | 30.4 |
| OSUM-Pangu | 44.6 | 28.9 | 29.5 |
Speech Foundation Models
Speech foundation models are large pretrained systems that learn reusable audio representations for speech understanding, recognition, and generation.
See all 65 papers