RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

Jaber Jaber; Osama Jaber

cs.CLApr 10, 2026

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

Authors: Jaber Jaber, Osama Jaber

Organizations: RightNow AI

Abstract

Open Arabic large language models split into two classes: sub-1B multilingual models that treat Arabic as an afterthought (Qwen2.5-0.5B, Falcon-H1-0.5B), and 7B-70B Arabic-specialized models that require a server to run (Jais, AceGPT, ALLaM, SILMA). The one published attempt at a sub-2B Arabic-specialized model, Kuwain-1.5B, never released its weights. We present RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder LLM built on Qwen2.5-0.5B. The pipeline adds 27,032 Arabic tokens via mean-subtoken initialization, continues pretraining on 504M Arabic tokens on 8xH100 with FSDP, FlashAttention varlen packing, and Liger fused kernels, then applies supervised fine-tuning on 129,116 Arabic instruction pairs with response-only loss masking, direct preference optimization on 6,750 Arabic preference pairs, and weight soup merging across three checkpoints. On three lm-evaluation-harness Arabic benchmarks (COPA-ar, Arabic HellaSwag, ArabicMMLU) the merged model reaches 35.9% mean accuracy, beats every same-class open model, ties Falcon-H1-1.5B on COPA-ar (58.4%) at one-third the size, and recovers 67% of SILMA-9B's mean at 1/18 the parameters. The edge build quantizes to 398 MB (q4_k_m) and delivers 635 tokens/s at batch size 1 on a single H100 via llama.cpp. All code (5,555 lines across 25 scripts), weights (bf16, int8, and four GGUF quantizations), and benchmark scripts are released at https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo.

Figures & tables

Model	Params	Open weights	Arabic-specialized
Qwen2.5-0.5B-Instruct ( Qwen Team et al., 2024 )	494M	✓	✗
Falcon-H1-0.5B-Instruct	524M	✓	✗
RightNow-Arabic-0.5B-Turbo (ours)	518M	✓	✓
Falcon-H1-1.5B-Instruct	1.5B	✓	✗
Kuwain-1.5B ( Hennara et al., 2025 )	1.5B	✗	✓
AceGPT-7B-chat ( Huang et al., 2023 )	7B	✓	✓

Table 1: The open Arabic LLM landscape. Our model is the only sub-1B Arabic-specialized decoder with publicly available weights.

Panel of Figure 1 — Figure 1: Training and deployment pipeline. Each stage produces a checkpoint that feeds the next; the soup stage also consumes the pretrain and SFT checkpoints directly.

Component	Lines	Notes
Phase scripts (00–07)	1,904	env check, tokenizer surgery, data, train, eval
Runtime modules src/rnar/	1,468	FSDP, FA patch, data loader, train loop, merger
Benchmark + merge scripts	1,047	competitor bench, merge variants, speed tests, GGUF
Pre-tokenization scripts	318	flat memmap producers for pretrain and SFT
Chart generation	524	matplotlib Pareto and per-task bar plots
Configs	171	YAML hyperparameters for pretrain, SFT, DPO

Table 2: Codebase metrics. Every training, benchmark, merge, and export step is reproducible from the released scripts.

Same-class open (0.5B)
Model	Params	COPA-ar	HellaSwag-ar	ArabicMMLU	Mean
Qwen2.5-0.5B-Instruct ( Qwen Team et al., 2024 )	494M	53.9%	22.5%	26.0%	34.1%
Falcon-H1-0.5B-Instruct	524M	44.9%	23.0%	24.2%	30.7%
Ours (soup_dpo_sft_pre)	518M	58.4%	26.0%	23.2%	35.9%
Larger open
Falcon-H1-1.5B-Instruct	1.5B	58.4%	27.5%	32.7%	39.5%

Table 3: Arabic benchmark results. Ours is the only Arabic-specialized open model at 0.5B class. Methodology: lm-eval-harness v0.4.11, apply_chat_template=True (fallback for AceGPT-7B-chat which has no template), limit=200 per task, acc_norm preferred. Bold indicates the best in column; the last column is the unweighted mean.

Panel of Figure 2 — Figure 2: Mean Arabic benchmark accuracy (COPA-ar, HellaSwag-ar, ArabicMMLU) versus parameter count. Our model is the only Arabic-specialized entry in the sub-1B edge-deployable zone. Dashed line shows the Pareto frontier.

Panel of Figure 3 — Figure 3: Per-task accuracy breakdown across all evaluated models. Our 0.5B model (dashed box) wins COPA-ar and HellaSwag-ar among same-class models. The ArabicMMLU gap to 7B+ models reflects the knowledge ceiling at sub-1B scale.

Checkpoint	COPA-ar	HellaSwag-ar	ArabicMMLU	Mean
soup (DPO 0.5, SFT 0.25, Pretrain 0.25)	58.43%	25.33%	23.17%	35.64%
lerp(DPO, Pretrain, t=0.3)	58.43%	24.67%	23.17%	35.42%
slerp(DPO, Pretrain, t=0.3)	58.43%	24.67%	23.15%	35.42%
lerp(DPO, Pretrain, t=0.5)	58.43%	24.67%	23.04%	35.38%
slerp(DPO, Pretrain, t=0.5)	58.43%	24.67%	22.98%	35.36%
lerp(DPO, Pretrain, t=0.7)	55.06%	28.00%	22.83%	35.29%

Table 4: Weight soup merge ablation. The 50/25/25 soup of DPO, SFT, and pretrain checkpoints wins on mean. Differences are small (0.44 points spread) but reliable: the best merge is chosen by downstream metric, not training loss. Small differences from Table 3 are due to a separate evaluation pass; Table 3 reports the final unified run.

Tokenizer	Vocab size	Tokens for sample	Fertility
Qwen2.5-0.5B baseline	151,665	803	2.18
Ours (with added 27,032 Arabic tokens)	178,697	664	1.80
Reduction	+27,032	$-139$	$-17.3\%$

Table 5: Arabic tokenizer fertility (tokens per word) on a 368-word held-out sample. Lower is better. Our merged tokenizer uses 17.3% fewer tokens per Arabic word than the Qwen2.5 baseline tokenizer.

Quantization	Disk	Prompt eval (tok/s)	Generation (tok/s)
f16	988 MB	634.0	582.4
q8_0	525 MB	732.8	645.7
q5_k_m	419 MB	718.5	633.5
q4_k_m	398 MB	723.6	634.9

Table 6: llama.cpp CUDA inference speed per GGUF quantization on a single H100 SXM5. All numbers from llama-bench -n 128 -p 128 -b 1 -ngl 99 . q8_0 is the best throughput while q4_k_m is the smallest footprint.

Phase	Steps	Start loss	End loss	Wall time
Pretrain (arwiki, 504M tokens)	2,500	14.21	1.69	6h 57m
SFT (129k instructions, 5 epochs, loss-masked)	418	1.95	1.81	12m
DPO (6.75k preference pairs, 2 epochs)	844	0.693	0.691	34m

Table 7: Training phase summary. Pretraining step counts start at 0 and the loss column shows first-logged vs final-logged batch loss (bf16, accumulated across micro-batches and averaged across ranks via all-reduce).

Panel of Figure 4 — Figure 4: Pretraining loss over 2,500 steps on 504M Arabic Wikipedia tokens. The steep initial drop (steps 0–200) corresponds to the model learning the 27,032 new Arabic token embeddings. Loss plateaus near 1.69 (perplexity 5.42) by step 2,000.

Explore similar work

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
Apr 7, 2026Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair +6Arabic NLP LLM Fine-Tuning
Forta, Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
Evaluation of Small Language Models for Arabic Language Processing
Jun 19, 2026Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5Arabic NLP Small Language Models
Naseej Innovation Lab, Naseej for Technology, Riyadh, Saudi Arabia.
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
Aug 8, 2026Ahmed Amine Aliane, Hassina Aliane, Nasredine SemmarLanguage Model Pretraining Arabic NLP
Arabic Institute for Translation, Algiers, Algeria · CERIST, Algeria · CEA LIST, Paris-Saclay, France

Apr 7, 2026J/K move · Enter open · S save

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation

Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair, Jyani Akshay Jagdishbhai+5

Forta, Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA

Arabic NLP LLM Fine-Tuning Low-Resource Language Processing Language Model Distillation CoT Distillation

Open paper AskPDF arXiv

This paper introduces Arabic-DeepSeek-R1, an application-driven open-source Arabic LLM that leverages a sparse MoE backbone to address the digital equity gap for under-represented languages, and establishes a new SOTA across the entire Open Arabic LLM Leaderboard (OALL). Our four-phase CoT distillation scheme integrates Arabic-specific linguistic verification and regional ethical norms into a 372M-token, contamination-controlled 80/20 Arabic-English training mixture. Arabic-DeepSeek-R1 achieves the highest average score across the seven-benchmark OALL suite while establishing SOTA or near-SOTA, including dominant results on grammar-focused MadinahQA (surpassing both GPT-5.1 and the OALL leader by substantial margins), safety-oriented AraTrust, multi-ability AlGhafa, and retrieval-augmented ALRAGE. Our results indicate that the combination of sparse MoE architecture, culturally-informed CoT distillation with explicit Arabic linguistic checks, and strategic bilingual data curation enables an open-source adapted model to systematically outperform the proprietary frontier system GPT-5.1 on the majority of benchmarks evaluating comprehensive language-specific tasks: the first such demonstration for Arabic LLMs. These findings indicate that much of Arabic's performance deficit in current LLM ecosystems stems from under-specialization rather than architectural limitations, and that parameter-efficient adaptation of open reasoning models can yield breakthrough SOTA performance without industrial-scale pretraining costs. Arabic-DeepSeek-R1 establishes a validated and replicable framework for sovereign and domain-specific language technologies, demonstrating that strategic, culturally-grounded adaptation of sparse MoE backbones offers a viable and cost-effective pathway to achieving record-breaking performance across standardized benchmarks for low-resource languages.

Abstract

Explore similar work

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation

Evaluation of Small Language Models for Arabic Language Processing

AraSSM: A bidirectional state-space encoder for Arabic masked language modeling