HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

Harish Santhanalakshmi Ganesan; Hariharan M. S.; Prabhurajhan R

cs.AIOct 25, 2026

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

Authors: Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R

Abstract

We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single "omni" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it.

Figures & tables

Panel of Figure 1 — Figure 1: Zero-shot average on seven tasks (HellaSwag, OpenBookQA, WinoGrande, ARC-c, ARC-e, BoolQ, PIQA) against pretraining tokens, for base checkpoints of up to about 1B parameters (HobbyLM-1B: 1.04B total, 305M active). Filled points were run through the same evaluation wrapper (Table 5 ). HobbyLM-1B (hollow) was evaluated by its builders on the same tasks with the same metric choices, but with lm-eval 0.4.13 rather than 0.4.9.1 (Appendix D ). HobbyLM-130M and -500M outscore MicroLlama-300M, OPT-350M and Pythia-160M, which saw 1.25–30 $\times$ more tokens; Pythia-410M (300B tokens) remains ahead of HobbyLM-500M.

Component	Content	Designed / built by
Architecture	MoE decoder, routing, balancing, normalization, optimizer split, presets, ablations	Harish Santhanalakshmi Ganesan
130M / 500M pretraining	FineWeb training, context extension, lm-eval wrapper	Harish Santhanalakshmi Ganesan
500M post-training	Chat (SFT+DPO), diffusion LM, SAE, multimodal omni model, function calling, computer use	Harish Santhanalakshmi Ganesan
Inference	GGUF export, Rust engine hobby-rs , desktop app hobby-chat , demo Space	Harish Santhanalakshmi Ganesan
HobbyLM-Image	DiT design, data curriculum, training, Rust port	Harish Santhanalakshmi Ganesan
HobbyLM-1B architecture	The 1B preset of Section 3	Adopted (Harish Santhanalakshmi Ganesan)

Table 1: Provenance of each component. “Adopted” means the component was taken from the HobbyLM architecture specification and code without modification.

Panel of Figure 2 — Figure 2: The HobbyLM decoder. Every layer is pre-norm. Layer 0 uses a dense SwiGLU FFN; all later layers use a sparse MoE with $E$ routed SwiGLU experts, top- $k$ sigmoid routing, an auxiliary-loss-free selection bias $b$ , and (in the 500M and 1B presets) one shared expert.

Preset	$d$	$L$	$n_{q}/n_{kv}\times d_{h}$	dense FFN	$E\times f$	top- $k$	shared	Total	Active
130M	512	12	$8/2\times 64$	1536	$32\times 192$	8	0	140.0M	62.1M
500M	768	16	$12/3\times 128$	2304	$36\times 320$	6	1	500.8M	169.0M
1B	1024	20	$16/8\times 128$	2816	$64\times 224$	8	1	1037.1M	305.0M

Table 2: HobbyLM presets.

G

is the dense-to-expert FFN width ratio. All presets use 1 dense leading layer, RMSNorm with

\epsilon=10^{-6}

, RoPE

\theta=10^{4}

(raised to

10^{6}

for the 8k-context 500M), tied embeddings and vocabulary 50,304.

Variant	Change from baseline	Val loss	$\Delta$
topk8	top-8 instead of top-4	3.6441	$-0.025$
no_renorm	no top- $k$ gate renormalization	3.6542	$-0.015$
softmax	softmax instead of sigmoid gating	3.6677	$-0.001$
shared1	add one shared expert	3.6678	$-0.001$
baseline	–	3.6688	0
scale_emb	multiply embeddings by $\sqrt{d}$	3.6775	$+0.009$

Table 3: Single-knob ablations at 130M, 3,000 steps, FineWeb validation loss (baseline 3.6688).

Option	ms/step	tokens/s	Peak GB	130M val loss
baseline	222.5	36,821	33.3	4.0739
fused_ce	209.2	39,166	26.4	4.0681
polar	224.8	36,445	33.3	4.0790
fused_ce + polar	210.1	38,988	26.4	4.0855
FP8 head	223.8	36,605	33.8	10.90 (no gradient)

Table 4: Throughput options on a 1B speed probe (one H100, micro-batch 8, sequence 1024) and quality on a 130M, 1,000-step run.

Model	Params	Tokens	Hella	OBQA	Wino	ARC-c	ARC-e	BoolQ	PIQA	Avg
SmolLM2-360M	360M	4T	56.41	38.00	59.35	38.48	68.22	62.08	71.49	56.29
Qwen3-0.6B	600M	36T	53.81	34.60	58.56	38.31	58.08	69.82	70.29	54.78
SmolLM2-135M	135M	2T	43.13	33.20	52.25	29.44	58.59	60.37	68.39	49.34
Gemma-3-270M	270M	6T	41.28	30.60	53.43	28.33	57.07	57.77	68.12	48.09
Pythia-410M	410M	300B	40.14	28.80	52.96	23.98	45.45	58.50	67.52	45.34
HobbyLM-500M	501M/169M	40B	41.56	29.80	51.30	22.44	42.76	50.98	69.53	44.05

Table 5: 0-shot results, all base (not instruction-tuned) checkpoints, all run through the same harness wrapper. Reference models: SmolLM2 ( Ben Allal et al., 2025 ) , Qwen3 ( Yang et al., 2025a ) , Gemma 3 ( Gemma Team et al., 2025 ) , Pythia ( Biderman et al., 2023 ) , OPT ( Zhang et al., 2022 ) , GPT-2 ( Radford et al., 2019 ) and MicroLlama ( keeeeenw, 2024 ) .

Modality	Frozen encoder	Tokens per input	Projector
Image (early)	SigLIP2 so400m/14 @384	729 $\times$ 1152	1.48M
Image (from joint11)	SigLIP2 so400m/16 @512	1024 $\times$ 1152	1.48M
Video	same as image, per frame	729 or 1024 per frame	shared
Sound	CLAP HTSAT	flattened HTSAT feature map, $\times$ 768	1.18M
Speech	Whisper-small encoder	750 $\times$ 1536 (2-frame stacking)	1.77M

Table 6: Frozen encoders and projectors of the omni model: SigLIP2 ( Tschannen et al., 2025 ; Zhai et al., 2023 ) , CLAP ( Wu et al., 2023 ) with an HTSAT backbone ( Chen et al., 2022 ) , and Whisper ( Radford et al., 2023 ) .

Run	Added
joint5	image, video, sound, speech (spoken QA), text
joint6	single-shot tool calls
joint7	text agent loop (query $\to$ call $\to$ result $\to$ answer) and spoken agent loop (TTS’d queries)
joint8	text must-call up-weighted
joint9	Nemotron agentic SFT (v2) and Nemotron VLM captioning ( NVIDIA, 2025a ) , streamed
joint10	OCR (rendered text $\to$ markdown)

Table 7: Joint co-training lineage of the omni model. Each run adds paths to the cycle and resumes from its predecessor.

Checkpoint	Benchmark (scoring)	Score
Vision stage 2 (384px, 2k ctx)	POPE accuracy / F1	50.0 / 66.7
	GQA (normalized exact match)	27.85
	VQAv2 val (normalized exact match)	37.29
joint12 (512px, 8k ctx)	GQA (strict / containment)	20.6 / 39.2
joint12 (512px, 8k ctx)	VQAv2 val (strict / containment)	28.2 / 47.0
Speech stage 1 (projector only)	LibriSpeech test-clean WER $\downarrow$	28.54

Table 8: Multimodal results of the 500M model. For the stage-2 model, GQA and VQAv2 use 2K-question subsets and POPE the full 9K set.

Model	Eval set	JSON	Name F1	Args	Param. halluc. $\downarrow$	Exact
Nemotron SFT, greedy	Nemotron (400)	93.2	82.2	51.8	3.8	38.5
+ grammar	Nemotron (400)	–	–	–	0.0	38.5
+ weighted loss + grammar	Nemotron (400)	94.3	83.4	52.9	0.6	39.5
Combined data + grammar	BitAgent	–	100.0	–	0.0	59.0
Combined data + grammar	Nemotron	–	–	–	–	37.3

Table 9: Function calling on held-out data (needle-style metrics, %). “Grammar” is grammar-constrained decoding.

Split	JSON	Name F1	Value acc.	Exact
Planning (multi-step goals)	96.5	94.7	–	82.6
Grounding (real application trees)	${\sim}96$	95.5	91	78.4
Grounding (synthetic screens)	100	90.7	88.6	72.5

Table 10: Held-out computer-use results of HobbyLM-Computer-Use (%).

Adopted unchanged	1B preset (Table 2 ); sigmoid top-8-of-64 routing with aux-loss-free bias ( $u=10^{-3}$ , frozen at 95%); shared expert; QK-norm; GQA; router z-loss $10^{-3}$ , balance loss $10^{-3}$ , final-logit z-loss $10^{-4}$ ; chunked cross-entropy; fp32 router; Muon/AdamW split and $\sqrt{B}$ learning-rate scaling; warmup–stable–decay schedule; shard format
Added by the 1B build	100B-token curated mixture; tokenizer study; stratified shard ordering; anneal phase on high-quality data; 4 $\times$ H200 DDP training configuration; checkpoint-backup automation; evaluation; data-order and router-precision analyses; instruction tuning; macOS computer-use adapter

Table 11: HobbyLM-1B: adopted from the architecture vs. added by the 1B build.

Source	Dataset	Shards	Tokens	Share	Phase
Educational web	FineWeb-Edu (shards 000001–000600)	600	60B	60%	Main
Filtered web	DCLM-baseline-1.0	150	15B	15%	Main
Code	codeparrot-clean	100	10B	10%	Main
Math	FineMath-4plus	100	10B	10%	Main
Anneal	Cosmopedia-v2 (3.5B) + FineMath-4plus (1.5B)	50	5B	5%	Anneal

Table 12: HobbyLM-1B training data.

Third	Shards	Edu	DCLM	Code	Math
1	317	63.09%	15.77%	10.73%	10.41%
2	317	63.09%	15.77%	10.41%	10.73%
3	316	63.29%	15.82%	10.44%	10.44%
Target		63.2%	15.8%	10.5%	10.5%

Table 13: Family mix in each third of the 1B main-phase shard order.

Hardware	4 $\times$ NVIDIA H200 SXM, one node
Parallelism	PyTorch DDP via torchrun ; gradient all-reduce only on the last micro-step
Sequence / micro-batch / accumulation	1,024 / 32 per GPU / 8
Tokens per optimizer step	1,048,576
Precision	bf16 autocast, fp32 master weights, TF32 matmuls, fp32 router logits
Loss	chunked cross-entropy; final-logit z-loss $10^{-4}$ ; router z-loss $10^{-3}$ ; balance loss $10^{-3}$
Compilation	whole-model torch.compile

Table 14: HobbyLM-1B training configuration.

Task	Metric	Final (step 95,367)
HellaSwag	acc_norm	43.66 $\pm$ 0.49
PIQA	acc_norm	67.85 $\pm$ 1.09
WinoGrande	acc	52.41 $\pm$ 1.40
ARC-Easy	acc_norm	54.76 $\pm$ 1.02
OpenBookQA	acc_norm	35.00 $\pm$ 2.14
ARC-Challenge	acc_norm	29.27 $\pm$ 1.33

Table 15: HobbyLM-1B base, 0-shot release evaluation (lm-eval 0.4.13, fp32).

\pm

is the lm-eval standard error.

Run	Data	Tokens	Average
Baseline	FineWeb	10B	42.97
Sequential loader	curated mix, consumed source by source	5B	40.70
Stratified order	curated mix, interleaved	5B	43.34

Table 16: 130M data-order experiments, 7-task average; the baseline is the 130M FineWeb-10B model of Table 5 .

Benchmark	Metric	Base	Instruct
HellaSwag	acc_norm	43.66	46.64
ARC-Easy	acc_norm	54.76	54.12
ARC-Challenge	acc_norm	29.27	28.41
PIQA	acc_norm	67.85	68.88
WinoGrande	acc	52.41	50.75
OpenBookQA	acc_norm	35.00	36.20

Table 17: HobbyLM-1B base vs. Instruct, 0-shot (lm-eval 0.4.13, fp32, %). The average is over the seven tasks of Table 5 .

Stage	Res.	Data	Steps
Base	512	CC3M (140K), CC12M (320K), OmniEdit (40K), MagicBrush (2K)	–
Continued	512	ImageNet (50K), Midjourney-v6 recap (1M), UltraEdit (100K), OmniEdit (100K); 20–25% edit replay	45K + 120K
Sharpness	1024	continued mix at 1024px	20K
Accuracy	1024	FLUX-Reason-6M (500K + 150K people), Midjourney (50K)	up to 100K per run

Table 18: HobbyLM-Image training curriculum. Data counts are the configured per-source quotas; step counts are the defaults of the corresponding training actions.

Panel of Figure 3 — Figure 3: $1024\times 1024$ samples from HobbyLM-Image (final checkpoint, CFG $\approx 5$ , about 100 Euler steps), downscaled. Left: scene prompts. Right: object prompts. The object panel shows typical failures in fine structure (bus, bicycle).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Location	Description
HobbyLM-Base	rootxhacker/HobbyLM-Base	500M base model, 8k context (Section 4 )
HobbyLM-Chat	rootxhacker/HobbyLM-Chat	SmolTalk SFT and UltraFeedback DPO
HobbyLM-Computer-Use	rootxhacker/HobbyLM-Computer-Use	function calling and accessibility-tree agent (Section 5.4 )
HobbyLM-Omni	rootxhacker/HobbyLM-Omni	joint12 omni checkpoint with vision and speech projectors (Section 5 )
HobbyLM-Diffusion	rootxhacker/HobbyLM-Diffusion	masked-diffusion chat model (Section 4.4 )
HobbyLM-SAE	rootxhacker/HobbyLM-SAE	top- $k$ SAE on layer 8 of HobbyLM-Base

Appendix

Table 19: Released artifacts. Hugging Face repositories are under https://huggingface.co/ .

	130M	500M	500M ctx-2k	500M ctx-8k
Initialization	random	random	500M	500M ctx-2k
Steps	9,537	38,147	2,000	–
Tokens per step	1,048,576	1,048,576	1,048,576	–
Tokens	${\approx}10$ B	${\approx}40$ B	${\approx}2.1$ B	–
Sequence length	1,024	1,024	2,048	8,192
RoPE $\theta$	$10^{4}$	$10^{4}$	$10^{4}$	$10^{6}$

Appendix

Table 20: Pretraining and context-extension configuration (8

\times

H100, DDP, torch.compile ).

Stage	Trainable / data	LR	Warmup	Micro	Steps
Chat SFT	full model; SmolTalk ( ${\approx}1.5$ M)	$2\times 10^{-5\dagger}$	150 †	8 †	28,000
Chat DPO	full model; UltraFeedback, $\beta=0.1$	$5\times 10^{-7}$	–	2 †	2,000 †
Diffusion chat SFT	full model; SmolTalk, sequence 1,024	$2\times 10^{-5}$	150 †	16	8,000
Vision stage 1	projector; LCS-558K	$10^{-3\dagger}$	100 †	32 †	1,500
Vision stage 2	model + projector; LLaVA-Instruct-150K	$2{\times}10^{-5}$ / $10^{-4}$	60 †	8	2,500
Sound stage 1 (1 $\times$ H100)	audio projector; Clotho	–	–	–	1,200

Appendix

Table 21: Post-training configuration. All runs use 8

\times

H100 unless noted. Micro-batch is per GPU.

Backbone	DiT, 16 layers, width 1024, 16 heads, MLP ratio 3, patch 1, 333M parameters
Input	two-panel DC-AE f32c32 latent plus 2 mask channels (34 channels)
Conditioning	CLIP ViT-L/14 text features (64 tokens $\times$ 768), timestep, task
Objective	rectified flow, logit-normal $t$ , foreground weight $1+1.5m$ for edits
Optimizer	AdamW, $\beta=(0.9,0.95)$ , weight decay 0.02, gradient clipping 1.0
Learning rate	$5\times 10^{-5}$ from scratch, $2\times 10^{-5}$ continued; warmup 1,000 / 500 steps; cosine to 10%
Micro-batch	32 (512px pilot), 64 (512px continued), 16 (1024px)

Appendix

Table 22: HobbyLM-Image configuration.

Run	Hardware	Wall-clock	GPU-hours / cost
130M pretraining (10B tokens)	8 $\times$ H100	${\approx}2$ h	${\approx}16$
500M pretraining (40B tokens)	8 $\times$ H100	${\approx}13$ h	${\approx}104$
500M context extension to 2k	8 $\times$ H100	${\approx}1$ h	${\approx}8$
Tool SFT (4,000 steps)	8 $\times$ H100	${\approx}23$ min	${\approx}3$
joint co-training (1,600 steps)	8 $\times$ H100	${\approx}40$ min	${\approx}5$
Diffusion LM adaptation ( ${\approx}20$ B tokens)	8 $\times$ B200	–	${\approx}$ $155

Appendix

Table 23: Recorded compute for the main runs. GPU-hours are wall-clock time multiplied by the number of GPUs; where only the cost was recorded, we give the cost.

Stage	Identity	Steps
Base (step 95,367)	947e80d9	pretraining
2K extension	0add7bf0	550
Function-calling pilot	3f60c420	100
4K plain-RoPE extension (R2)	4a0c8215	150
SFT D: recovery, step 500	7f1e7213	500
SFT E: continuation, step 750	df65ea5a	250

Appendix

Table 24: Selected checkpoint path of HobbyLM-1B-Instruct. Identity is the SHA-256 prefix of each stage’s output. The 800 extension and pilot steps precede the SFT step counter. The step-2,751–4,750 continuation and a separate step-3,400 summarization branch are not ancestors of step 3,450.

SFT stage	Supervised tokens
D, step 0 $\rightarrow$ 500	11,869,621
E, step 500 $\rightarrow$ 750	5,789,137
F, step 750 $\rightarrow$ 2,750	43,678,658
G, step 2,750 $\rightarrow$ 3,450	16,776,587
Cumulative	78,114,003

Appendix

Table 25: Supervised-token increments (SFT plus replay) recorded in the checkpoints. The final raw-position counter is

3{,}617{,}587{,}200=3{,}450\times 1{,}048{,}576

, which includes padded and masked positions.

SFT bucket	Source	Rows	Share
Nemotron Policy-B	nvidia/Nemotron-SFT-Agentic-v2 ; consumed file’s revision unknown	130,411	22.6%
Verified math v2	internal generation	159,852	11.6%
Math answer-key v1	internal generation	120,000	8.7%
smol-magpie-ultra	SmolTalk, revision 5feaf2fd	261,204	8.5%
Tool trajectory v1	internal generation	30,000	7.5%
Python coding v2	internal generation	89,998	6.0%

Appendix

Table 26: Prepared datasets and configured sampling shares for the final Broad-Instruct stage (step 2,750

\rightarrow

3,450). Shares are conditional on selecting an SFT row and are rounded; replay is a separate share of about 0.46% of all sampled rows. Row counts are prepared rows, not unique examples consumed. Earlier stages used different mixtures and, for some buckets, different files. Revision prefixes are as recorded.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-of-Experts Language Models Efficient Language Model Training
The School of AI Bengaluru, India
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI

Apr 28, 2026J/K move · Enter open · S save

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi+4

Alibaba International Digital Commerce

Multilingual Language Models Mixture-of-Experts Language Models Activation Sparsity Efficient Language Model Training Open-Weight Language Models Sparse Neural Networks

Open paper AskPDF arXiv

We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-Instruct variants, which surpass the performance of competing models possessing

3

14\times

more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.

Abstract

Mixture-of-Experts Language Models

Explore similar work

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

Speculating Experts Accelerates Inference for Mixture-of-Experts