JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency

Aichen Cai; Anmeng Zhang; Anyu Li; Bo Zhang; Bohua Cai; Chang Li; Changjian Jiang; Changkai Lu

cs.CLApr 3, 2026

JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency

Authors: Aichen Cai, Anmeng Zhang, Anyu Li, Bo Zhang, Bohua Cai, Chang Li, Changjian Jiang, Changkai Lu, Chao Xue, Chaocai Liang, Cheng Zhang, Dongkai Liu, Fei Wang, Guoqiang Huang, Haijian Ke, Han Lin, Hao Wang, Ji Miao, Jiacheng Zhang, Jialong Shi, Jifeng Zhu, Jingjing Qian, Junhui Luo, Junwu Xiong, Lam So, Liang Huang, Ming Ke, Mingyang Li, Panfeng Shi, Peng Hao, Qi Wang, Qian Lai, Qiaoqiao Yuan, Qingyu Yin, Qiong Cao, Qixiang Wang, Rongcheng Bian, Rongduo Han, Shaoqiang Zheng, Shi Hu, Shi Suo, Shijie Ren, Shijin Zhang, Shiying Fan, Shuai Xie, Tianyi Zhang, Wei Liu, Wentao Tan, Xianghan Meng, Xiaodong He, Xing Pan, Xiran Wang, Xuyang Peng, Ya Zhang, Yang Liu, Yangyang Duan, Yanxu Chen, Yicheng Gong, Yidan Huang, Yifei Liu, Yinhao Bai, Yongqiang Liu, Yuesong Zhang, Yuqi Zhang, Zerui Xie, Zhenfang Wang, Zhennan Shen, Zheyuan Liu, Zhuwei Zeng

Abstract

We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances thinking and non-thinking cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.

Figures & tables

Figure 1: Model performance vs. token consumption across different middle-scale LLMs. The accuracy and token consumption averaged across eighteen benchmarks used in post-training evaluation (Table 3 ) are illustrated, where the upper-right region indicates more token-efficient models. Bubble size represents the model parameter count.

Hyperparameter	JoyAI-LLM Flash 48B-A3B
General Settings
Total Layers ( $N_{\text{layers}}$ )	40
Dense Layers	1
Hidden Dimension ( $d_{\text{model}}$ )	2048
Vocabulary Size ( $\|V\|$ )	129K
Max Context Length	128K

Table 1: Detailed architectural configurations of JoyAI-LLM Flash.

Figure 2: Agentic trajectory synthesis pipeline

Panel of Figure 3 — Figure 3: Scaling laws for JOYAI-LLM Flash. The plot illustrates the relationship between training compute and model performance. Data points represent empirical observations, while solid lines indicate the power-law fits.

Task	Qwen3-30B-A3B-Base	Qwen3.5-35B-A3B-Base	JoyAI-LLM Flash-Base
General Knowledge
MMLU	82.1	88.4	84.7
MMLU-Pro	61.7	60.7	73.1
CMMLU	83.6	86.1	83.1
Math
GSM8K	90.3	90.5	88.7

Table 2: Comparison of Base Model between Qwen3-30B-A3B, Qwen3.5-35B-A3B and JoyAI-LLM Flash. Best results are marked in bold.

Figure 4: Verifiable Environment Pipeline

Figure 5: (a) Aggregate gate $g^{\rm agg}$ (Eq. 4 ) with three regimes: pass-through ( $|x|\leq C$ , slope $1$ ), rollback ( $C<|x|<C^{*}:=(1+T_{\tau}^{-1})C$ , slope $-T_{\tau}$ ), and zeroed ( $|x|\geq C^{*}$ , output $0$ ). As $T_{\tau}$ increases, the rollback zone narrows (width $C/T_{\tau}$ ) and $g^{\rm agg}$ approaches a hard clip at $\pm C$ . (b) Base weight $\log w_{\tau}^{\rm base}$ (Eq. 2 ) in $(\log s^{+},\log s^{-})$ -space with asymmetric thresholds. Dashed lines mark the budget boundaries $C^{\pm}$ (onset of rollback), and dotted lines mark the full-gating thresholds $C^{*\pm}$ (onset of zeroing). The five global regimes follow a non-monotonic pattern: $|\log w|$ rises through the rollback onset (G-II,r), peaks when one channel is fully gated (G-II), declines under mutual rollback (G-III,r), and collapses to zero when both channels are fully gated (G-III, $w^{\rm base}_{\tau}=1$ ).

Figure 6: Single-domain RLVR on DAPO-Math-17k [ 72 ] : (a) training reward and (b) validation accuracy (AIME 2024 mean@1) vs. training step. GRPO collapses after step 60. GSPO stagnates. FiberPO improves steadily on both metrics.

Figure 7: Training diagnostics for the single-domain DAPO math run. Top row (comparative, all three methods): (a) policy entropy, (b) mean importance ratio on log scale, (c) mean response length. Bottom row (FiberPO-specific): (d) gradient norm on log scale, (e) fiber residual and token-level clip fraction (both in the safe zone throughout training).

Panel of Figure 8 — Figure 8: Comparison of model performance (left bars) and token consumption (right bars) across six benchmarks. Qwen3.5-35B-A3B and JoyAI-LLM Flash are instruct models, “GLM-4.7-Flash-T” refers to GLM-4.7-Flash-Thinking, which is included due to its comparable performance.

Task	Qwen3-Next-80B-A3B		Qwen3.5-35B-A3B		Qwen3-30B-A3B		GLM-4.7-Flash-T		JoyAI-LLM Flash
Task	Acc	#Token	Acc	#Token	Acc	#Token	Acc	#Token	Acc	#Token
General Knowledge
MMLU	86.4	400	91.2	900	88.0	200	88.2	2000	89.1	200
MMLU-Pro	76.7	1500	81.6	4000	78.7	1600	75.8	6200	81.6	900
HellaSwag	88.1	¡100	89.1	¡100	86.2	¡100	71.5	3000	91.7	¡100
CMMLU	89.1	500	89.5	600	87.1	500	80.2	4900	86.7	400

Table 3: Comparison with baseline models. Qwen3-Next-80B-A3B, Qwen3.5-35B-A3B, and Qwen3-30B-A3B are instruct models; “GLM-4.7-Flash-T” refers to GLM-4.7-Flash-Thinking, which is additionally included due to its more comparable performance. Results marked with asterisk ∗ are directly cited from original papers and differ substantially from our reproduced results.

Figure 9: Comparison of Accuracy and Throughput for Quantized Models: JoyAI-LLM Flash vs. Qwen3-30B-A3B. The accuracy is measured by the mean accuracy across three distinct domain datasets: MATH-500, GPQA, and MBPP.

Model	GPU	Speedup	Acceptance		TPS
Model	GPU	Speedup	length	ratio	user	server
JoyAI-LLM Flash	1	1.87×	2.20	40.35	66	4241
GLM-5 [ 95 ]	8	1.82×	3.03	75.84	31	1969
DeepSeek-V3.2 [ 36 ]	8	1.79×	2.55	63.72	31	1958
Qwen3.5-35B-A3B [ 6 ]	1	1.61×	3.18	72.56	85	5428
MiMo-V2-Flash [ 15 ]	4	1.61×	2.69	67.22	47	3033

Table 4: SpecBench MTP-3 Speculative Decoding Performance. Best results are marked in bold.

Figure 10: Joint Optimization of MTP and Quantization (ISL/OSL = 1K/2K, Concurrency = 64)

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget
Oct 25, 2026Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan RMixture-of-Experts Language Models Efficient Language Model Training
TokenRouter: Efficient Serving System for Token-Level LLM Routing
Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM Serving LLM Routing
Tsinghua University · Carnegie Mellon University
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference

Oct 25, 2026J/K move · Enter open · S save

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R

Mixture-of-Experts Language Models Efficient Language Model Training T2I Generation Multimodal Large Language Models Diffusion Transformer

Open paper AskPDF arXiv

We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single "omni" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it.

Abstract

Language Modeling

Explore similar work

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

TokenRouter: Efficient Serving System for Token-Level LLM Routing

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence