We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances thinking and non-thinking cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.
Figures & tables
Figure 1: Model performance vs. token consumption across different middle-scale LLMs. The accuracy and token consumption averaged across eighteen benchmarks used in post-training evaluation (Table 3 ) are illustrated, where the upper-right region indicates more token-efficient models. Bubble size represents the model parameter count.
Hyperparameter
JoyAI-LLM Flash 48B-A3B
General Settings
Total Layers ( Nlayers )
40
Dense Layers
1
Hidden Dimension ( dmodel )
2048
Vocabulary Size ( ∣V∣ )
129K
Max Context Length
128K
Table 1: Detailed architectural configurations of JoyAI-LLM Flash.
Figure 2: Agentic trajectory synthesis pipeline
Figure 3: Scaling laws for JOYAI-LLM Flash. The plot illustrates the relationship between training compute and model performance. Data points represent empirical observations, while solid lines indicate the power-law fits.
Task
Qwen3-30B-A3B-Base
Qwen3.5-35B-A3B-Base
JoyAI-LLM Flash-Base
General Knowledge
MMLU
82.1
88.4
84.7
MMLU-Pro
61.7
60.7
73.1
CMMLU
83.6
86.1
83.1
Math
GSM8K
90.3
90.5
88.7
Table 2: Comparison of Base Model between Qwen3-30B-A3B, Qwen3.5-35B-A3B and JoyAI-LLM Flash. Best results are marked in bold.
Figure 4: Verifiable Environment Pipeline
Figure 5: (a) Aggregate gate gagg (Eq. 4 ) with three regimes: pass-through ( ∣x∣≤C , slope 1 ), rollback ( C<∣x∣<C∗:=(1+Tτ−1)C , slope −Tτ ), and zeroed ( ∣x∣≥C∗ , output 0 ). As Tτ increases, the rollback zone narrows (width C/Tτ ) and gagg approaches a hard clip at ±C . (b) Base weight logwτbase (Eq. 2 ) in (logs+,logs−) -space with asymmetric thresholds. Dashed lines mark the budget boundaries C± (onset of rollback), and dotted lines mark the full-gating thresholds C∗± (onset of zeroing). The five global regimes follow a non-monotonic pattern: ∣logw∣ rises through the rollback onset (G-II,r), peaks when one channel is fully gated (G-II), declines under mutual rollback (G-III,r), and collapses to zero when both channels are fully gated (G-III, wτbase=1 ).
Figure 6: Single-domain RLVR on DAPO-Math-17k [ 72 ] : (a) training reward and (b) validation accuracy (AIME 2024 mean@1) vs. training step. GRPO collapses after step 60. GSPO stagnates. FiberPO improves steadily on both metrics.
Figure 7: Training diagnostics for the single-domain DAPO math run. Top row (comparative, all three methods): (a) policy entropy, (b) mean importance ratio on log scale, (c) mean response length. Bottom row (FiberPO-specific): (d) gradient norm on log scale, (e) fiber residual and token-level clip fraction (both in the safe zone throughout training).
Figure 8: Comparison of model performance (left bars) and token consumption (right bars) across six benchmarks. Qwen3.5-35B-A3B and JoyAI-LLM Flash are instruct models, “GLM-4.7-Flash-T” refers to GLM-4.7-Flash-Thinking, which is included due to its comparable performance.
Task
Qwen3-Next-80B-A3B
Qwen3.5-35B-A3B
Qwen3-30B-A3B
GLM-4.7-Flash-T
JoyAI-LLM Flash
Acc
#Token
Acc
#Token
Acc
#Token
Acc
#Token
Acc
#Token
General Knowledge
MMLU
86.4
400
91.2
900
88.0
200
88.2
2000
89.1
200
MMLU-Pro
76.7
1500
81.6
4000
78.7
1600
75.8
6200
81.6
900
HellaSwag
88.1
¡100
89.1
¡100
86.2
¡100
71.5
3000
91.7
¡100
CMMLU
89.1
500
89.5
600
87.1
500
80.2
4900
86.7
400
Table 3: Comparison with baseline models. Qwen3-Next-80B-A3B, Qwen3.5-35B-A3B, and Qwen3-30B-A3B are instruct models; “GLM-4.7-Flash-T” refers to GLM-4.7-Flash-Thinking, which is additionally included due to its more comparable performance. Results marked with asterisk ∗ are directly cited from original papers and differ substantially from our reproduced results.
Figure 9: Comparison of Accuracy and Throughput for Quantized Models: JoyAI-LLM Flash vs. Qwen3-30B-A3B. The accuracy is measured by the mean accuracy across three distinct domain datasets: MATH-500, GPQA, and MBPP.
Model
GPU
Speedup
Acceptance
TPS
length
ratio
user
server
JoyAI-LLM Flash
1
1.87×
2.20
40.35
66
4241
GLM-5 [ 95 ]
8
1.82×
3.03
75.84
31
1969
DeepSeek-V3.2 [ 36 ]
8
1.79×
2.55
63.72
31
1958
Qwen3.5-35B-A3B [ 6 ]
1
1.61×
3.18
72.56
85
5428
MiMo-V2-Flash [ 15 ]
4
1.61×
2.69
67.22
47
3033
Table 4: SpecBench MTP-3 Speculative Decoding Performance. Best results are marked in bold.
Figure 10: Joint Optimization of MTP and Quantization (ISL/OSL = 1K/2K, Concurrency = 64)
We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single "omni" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it.
Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.
We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-Constrained Hyper-Connections (mHC) that enhance conventional residual connections; (3) and the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible. The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4.