TeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks
Abstract
We present Xing 4, a series of large language models that extends the TeleChat lineage through coordinated advances in model architecture and training infrastructure on Ascend. This report describes the design principles and systems engineering behind the series, spanning architecture selection, large-scale pretraining, and reasoning and agentic post-training. We study how sparse activation, attention design, model depth and width, and speculative decoding balance model capacity with computation, memory access, and execution overhead. Our pretraining infrastructure builds on MindSpore HyperParallel, combining topology-aware hybrid sharding, efficient distributed optimizer execution, and coordinated management of long-sequence workloads and activation memory. CPU affinity and core isolation further reduce host scheduling interference. Together, these mechanisms support the pretraining of a 285B-parameter model on 8,192 Ascend 910C cards, providing a concrete setting for the architectural and systems studies. For post-training, we introduce TeleGym, a unified agent execution platform that connects execution, evaluation, and trajectory construction to policy optimization. Disaggregated asynchronous execution, bounded oversampling, and partial rollout accommodate heterogeneous task durations while preserving trajectory continuity across policy updates. Together, these advances establish an Ascend-oriented training foundation for the continued development of Xing 4 across model scales and increasingly demanding reasoning and agent workloads.
Figures & tables
| Item | Configuration |
| Inference environment | 32 Ascend 910C cards |
| Deployment mode | P/D disaggregation |
| Prefill-side parallelism | TP4DP4 |
| Decode-side parallelism | TP2DP8 |
| Evaluation context length | 256K |
| Architecture | KV HBM Access | Index Computation | Relative TTFT | Relative TPOT | Prefill Throughput | Decode Throughput |
| MLA | 1.00 | None | 1.00 | 1.00 | 1.00 | 1.00 |
| MLA + DSA | 0.85 | Per-DSA-layer | 0.37 | 0.84 | 2.67 | 1.18 |
| MLA + DSA + IndexShare | 0.80 | Cross-layer | 0.21 | 0.79 | 4.68 | 1.27 |
| Configuration | Layers | Hidden Size | Heads | Routed Experts | Expert Intermediate Size | Experts per Token | Shared Experts |
| Baseline | 60 | 4096 | 128 | 256 | 1536 | 6 | 1 |
| Wider-shallower | 45 | 4096 | 128 | 256 | 2048 | 8 | 1 |
| Configuration | Relative TTFT | Relative TPOT | Prefill Throughput | Decode Throughput | Decision |
| Baseline | 1.00 | 1.00 | 1.00 | 1.00 | Reference |
| Wider-shallower | 0.83 | 0.90 | 1.20 | 1.11 | Selected |
| Benchmark | Output Throughput (tokens/s) | MTP3/Base | DSpark/Base | DSpark/MTP3 |
| GSM8K | 35.4 / 63.7 / 115.8 | |||
| MATH-500 | 36.8 / 70.0 / 128.1 | |||
| HumanEval | 35.7 / 64.1 / 134.7 | |||
| MBPP | 37.1 / 70.0 / 132.0 | |||
| AIME25 | 37.0 / 71.5 / 109.2 | |||
| LiveCodeBench | 35.6 / 64.5 / 95.2 |
| Model | Layers | Hidden Size | Routed Experts | Expert Intermediate Size | Experts per Token | Shared Experts |
| Xing 4 (29B) | 40 | 3584 | 64 | 1024 | 4 | 1 |
| Xing 4 (285B) | 45 | 4096 | 256 | 2048 | 8 | 1 |
| Scheme | Pipeline | DP (Replica Shard) | mbs | Step Time | Speedup |
| PP-1F1B (Baseline) | |||||
| PP-Interleaved | |||||
| HSDP ( Ours ) |
| Configuration | Step Time | Speedup | Step-time reduction |
| HSDP | — | ||
| HSDP + owner-based Muon |
| Kernel | Duration (ms) | Output size (MiB) |
| SparseLightningIndexerGradKLLoss | 38.32 | 193.0 |
| LightningIndexer | 21.51 | 48.0 |
| AllGatherMatmul | 15.90 | 656.0 |
| SparseFlashAttention | 13.47 | 516.0 |
| MatmulReduceScatter | 6.00 | 64.0 |
| Configuration | Step Time | Speedup | Step-time reduction |
| HSDP + Muon | — | ||
| HSDP + Muon + selective recomputation |
| Configuration | Step Time | Speedup | Step-time reduction |
| HSDP + Muon | — | ||
| HSDP + Muon + layer-wise indexer backward |
| CPU contention | Critical threads wait | Work is submitted late | Steps take longer | |
| Training threads compete for CPU time with each other and with external work. | Data processing and task scheduling stall. | Compute dispatch and participation in communication are delayed. | Throughput falls or varies across steps. | |
| CPU pinning | Core isolation | |||
| Allocate training CPUs through a shared device pool or role-specific subsets. | Move selected system services and movable IRQ handling off training CPUs. | |||
| Training CPU set T per-device pool | ||
| Device 0: {8, 10, 12, 14, 16–39} 28 logical CPUs | ||
| Coarse-grained pinning | Fine-grained pinning | System CPU pool H |
| Shared by all roles Main, PyNative, MindData, runtime and support threads use all 28 CPUs. Load balancing remains enabled within the pool. | Pools by role PyNative: 8, 10, 12, 14 MindData: 16–23 Main, runtime and support threads: 24–39 (shared) | Services, logs, monitoring and movable IRQs Isolation on: restrict to H. Isolation off: restore recorded defaults. |
| 1. Fix conditions | 2. Set pinning | 3. Verify placement | 4. Run six configurations | 5. Assess effects | ||
| CPU budget, model, load and versions | Job, device or role level | Masks, cgroup and effective cpuset | Each policy with isolation off and on | Throughput, tail latency and scheduling delay | ||
| If verification fails, correct the constraints and repeat the check. If it passes, proceed to the six runs. Repeat tests to check attribution and retain results that show no improvement. | ||||||
| Baseline: job set T | Coarse: device pool | Fine: role subsets of | ||||
| B0: isolation off B1: isolation on | C0: isolation off C1: isolation on | F0: isolation off F1: isolation on | ||||
| Optimization | Cluster size | Throughput gain | Evaluation condition |
| CPU resource isolation | Small (512 accelerators) | 1–2% | Pinning and isolation combined |
| CPU resource isolation | Large (8,192 accelerators) | 5–10% | Pinning and isolation combined |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 376 papers