Hunyuan-A13B Technical Report

Tencent Hunyuan Team; Ao Liu; Botong Zhou; Can Xu; Chayse Zhou; ChenChen Zhang; Chengcheng Xu; Chenhao Wang

cs.AISep 23, 2026

Hunyuan-A13B Technical Report

Authors: Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang, Qian Wang, Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang, Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang, Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du, Zhen Yang, Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong, Yiqi Chen

Abstract

We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion during inference, balancing model capability, computational efficiency, and deployment cost. The model is pretrained on a rigorously filtered 20T-token corpus with enhanced STEM data curation, improving factual reliability and reasoning ability. High-quality supervised fine-tuning and large-scale reinforcement learning further enhance its overall performance. Hunyuan-A13B also introduces a dual-mode Chain-of-Thought framework that adapts reasoning depth to task complexity: fast thinking for routine queries and slow thinking for complex, multi-step problems. Evaluations show competitive performance across mathematics, science, programming, general language understanding, and agent tasks, often approaching that of much larger models. Its high inference throughput makes it suitable for latency-sensitive applications. We release Hunyuan-A13B to support open research and practical LLM deployment.

Figures & tables

Configuration	Hunyuan-A13B
# Layers	32
# Attention Heads	32
# Key/Value Heads	8
# Shared Experts	1
# Specialized Experts	64
# Activated Specialized Experts	8

Table 1: Overview of the key hyper-parameters of Hunyuan-A13B.

Figure 1: A diagram illustrating the four steps of Hunyuan-A13B post-training.

Model	Hunyuan-Large-1116	Qwen2.5-72B	Qwen3-A22B	Hunyuan-A13B
Architecture	MoE	Dense	MoE	MoE
# Activated Params	52B	72B	22B	13B
# Total Params	389B	72B	235B	80B
Context Length	256K	128K	128K	256K
General Tasks
MMLU	88.40	86.10	87.81	88.17

Table 2: Performance of Hunyuan-A13B and other representative MoE and Dense base models. The highest scores are shown in bold.

		OpenAI-o1-1217	Deepseek-R1-0120	Qwen3-A22B	Hunyuan-A13B
Mathematics	AIME2024	74.3	79.8	85.7	87.3
	AIME2025	79.2	70	81.5	76.8
	MATH	96.4	94.9	94	94.3
Science	GPQA-Diamond	78	71.5	71.1	71.2
	OlympiadBench	83.1	82.4	85.7	82.7
Coding	LiveCodeBench	63.9	65.9	70.7	63.9

Table 3: Comparison of Hunyuan-A13B with other AI models evaluating performance on slow-thinking mode. The highest and second-best scores are shown in bold and underlined, respectively.

		Hunyuan-Large-1116	Qwen2.5-72B-instruct	Qwen3-A22B	Hunyuan-A13B
Mathematics	AIME2024	23.3	18.9	40.1	30.6
	AIME2025	8.3	15	24.7	19.2
	MATH	77.4	83.1	87.2	85.4
Science	GPQA-Diamond	42.4	49	62.9	61.8
	OlympiadBench	44.8	52.9	69.9	64.1
Coding	LiveCodeBench	18.1	25.4	35.3	27.4

Table 4: Comparison of Hunyuan-A13B with other AI models evaluating performance on fast-thinking mode. The highest and second-best scores are shown in bold and underlined, respectively.

Model	Gemini 2.5 Pro	DeepSeek R1	Qwen3-A22B	Hunyuan-A13B
PenguinScrolls	88.3	87.5	87.1	87.7
LongBench-v2	60.9	53.8	48.4	55.0
FRAMES	80.1	85.7	84.0	81.1

Table 5: Performance comparison of Hunyuan-A13B and other representative open-source models on key long-context benchmarks. The highest scores are bolded.

Model	RULER
Model	Avg.	0-8K	8K-32K	32K-64K	64K-128K
Gemini 2.5 Pro	81.7	83.2	80.0	83.3	80.2
DeepSeek R1	72.0	75.4	72.1	72.1	65.6
Qwen3-A22B	73.0	76.6	72.8	72.9	66.6
Hunyuan-A13B	76.7	78.7	75.3	78.0	73.9

Table 6: Performance comparison of Hunyuan-A13B and other representative open-source models on the RULER benchmark’s QA task. This comparison demonstrates the models’ long-context performance across various input lengths, specifically highlighting Hunyuan-A13B’s effectiveness as context length increases.

Batch	Input Length	Output Length	Throughput (tokens/s)
1	2048	14336	190.84
16	2048	14336	1246.54
32	2048	14336	1981.99
32	2048	22528	1725.95

Table 7: Model Throughput of Hunyuan-A13B

Adaptive Reasoning in Language Models

See all 13 papers

Instella-MoE Technical Report
Sep 1, 2026Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra +10Mixture-of-Experts Language Models Language Model Post-Training
Motif 3: Technical Report
Aug 10, 2026Junghwan Lim, Joon Son Chung, Sungmin Lee +24Mixture-of-Experts Language Models Long-Context Language Modeling
Motif Technologies
HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning
Oct 2, 2026Donggyun Kim, Jack Lu, Chanwoo Kim +2Efficient Language Model Reasoning LLM Inference Acceleration
KAIST School of Computing · NYU Courant Institute School of Mathematics, Computing, and Data Science

Sep 1, 2026J/K move · Enter open · S save

Instella-MoE Technical Report

Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler+9

Mixture-of-Experts Language Models Language Model Post-Training LLM Training

Open paper AskPDF arXiv

In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research.

Abstract

Adaptive Reasoning in Language Models

Explore similar work

Instella-MoE Technical Report

Motif 3: Technical Report

HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning