Agaram 2.5B: The First Tamil-First Language Model Trained from Scratch with Tool Calling. A Playbook for Under-Represented Languages

Ramchand Kumaresan; Madhusudhanan G; Kaushik Kezendirane

cs.CLOct 13, 2026

Agaram 2.5B: The First Tamil-First Language Model Trained from Scratch with Tool Calling. A Playbook for Under-Represented Languages

Authors: Ramchand Kumaresan, Madhusudhanan G, Kaushik Kezendirane

Organizations: Murai Labs · Independent researcher

Abstract

We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, math, code and parallel text) on one node of eight H100 GPUs in about six days, with a new tokenizer that, unlike our first model's, keeps line breaks. Supervised fine-tuning on 134,428 conversations, whose replies an open-weight teacher model wrote, and three short targeted rounds turned it into a chat model that answers in Tamil script, romanized Tamil (Tanglish) and English and can call tools: on 25 tool prompts whose schemas appear in training it picks the right tool 23 times and gives exactly the expected arguments 13 times. The released model translates better than Sarvam-1, a 2B Indic base model, in all six FLORES and IN22 directions (FLORES chrF 54.3 into Tamil and 55.2 into English, against 49.2 and 44.2), and nearly doubles its base model on native-Tamil IndicIFEval (19.3% from 10.0%). It is weak on knowledge: reading-comprehension and knowledge benchmarks sit near chance, and on 50 held-out everyday questions written by one native speaker, one LLM judge rated 46% of its replies usable (22-24 of 50 over three passes; 95% interval for 23 of 50, 33-60%), with nearly every failure a wrong fact. Our two further SFT experiments changed answer format but not the knowledge measures. We also describe our first model, Agaram 1.5B, whose failures shaped the choices here; how the judge's scores varied between passes on identical replies; and, item by item, what the targeted rounds trained against and where frontier-model text entered the data. The project ran on compute credits and our own funds, about USD 13,300 across both models; the weights are released (licence to be announced after legal review), with GGUF files for llama.cpp, at https://huggingface.co/murai-labs/agaram-2.5b-instruct; code at https://github.com/Murai-Labs/Agaram2.5b

Figures & tables

Panel of Figure 1 — Figure 1: How \tamilfont அகரம் / Agaram 2.5B was built, from corpus to released model (details in Sections 3 – 6 ). Tokens per word are for Tamil. Shares are of the 50B training tokens, which repeat the scarcer Tamil sources; the other 22% is math (10%), code (7%), parallel text (2%) and instruction data (3%).

Figure 2: (a) The project from its first code to the released 2.5B checkpoint; data collection began earlier, in December 2025. (b) The 2.5B run in detail; bars are training time only (the SFT bars exclude evaluation and copying).

\tamilfont அகரம் / Agaram 1.5B	\tamilfont அகரம் / Agaram 2.5B
The tokenizer’s normaliser erased every newline and indentation; the model never saw a line break, and the chat stop token never appeared in pretraining.	A byte-level tokenizer with no normaliser (Section 3.1 ).
75% of the 30B tokens were single-pass Tamil web crawl; English, math and code were thin, and instruction-shaped text was 0.7%.	A measured 50B-token mix: 58% Tamil, the rest English, math, code and parallel text, with instruction data in the decay stage (Section 3.2 ).
Continued pretraining packed documents with no separators, so free chat ran on into memorised instruction text.	Every document ends with an end-of-text token, and attention never crosses a document (Section 4.1 ).
OCR damage explained several failures, and an OCR filter judged only on clean text discarded good Tamil 37 times in 41.	Narrow OCR rules, each checked by reading what it removes (Section 3.2 ).
The final checkpoint was lost when the node shut itself down.	Checkpoints copied off the node every two hours; the node releases itself only after the final copy (Section 4.3 ).
Benchmark translations and story openings used in evaluation were found in training data after numbers had been reported.	Every evaluation set scanned against the training data before its numbers are reported (Section 7 ).

Table 1: Problems in the first model and the corresponding choice in the second.

Sample	TOK_FRESH (this work)	TOK_BPE_64K ( \tamilfont அகரம் / Agaram 1.5B)	Qwen2.5
Tamil prose	1.55	1.55	9.44
Tamil verse	2.29	1.80	9.19
English	1.41	1.67	1.35
Tanglish	1.95	2.95	2.44
Code (Stack-Edu)	3.54	4.54	3.01
Tool-call JSON	2.52	3.46	2.35

Table 2: Tokens per whitespace-separated word (lower is cheaper). TOK_FRESH keeps every byte; the old tokenizer round-trips nothing that spans more than one line (code 0/320, verse 0/204).

Category	Share	Main sources
Tamil	58.2%	FineWeb2, Sangraha, CulturaX, HPLT, OSCAR, Wikipedia, Project Madurai, OCR of books
English	19.8%	FineWeb-Edu, Nemotron-CC (training only)
Math	10.0%	FineMath
Code	7.0%	Stack-Edu, Nemotron code (training only)
Parallel	2.0%	BPCC, FineTranslations, small parallel sets
Instruction	3.0%	decay stage only (23% of its tokens)

Table 3: Planned share of the 50B training tokens by category.

Figure 3: Stable-stage mix by source (43.5B tokens including repeats, 63 sources).

Figure 4: Share of each stage’s tokens for the 16 largest sources. Decay drops the broad crawls and adds task and instruction data in plain-text chat form.

Panel of Figure 5 — Figure 5: The decoder block, repeated 28 times (pre-norm; teal lines are the residual connections), and the parameter count read from the released weights’ tensor shapes.

	Stable	Decay
Context (tokens)	4,096	8,192
Sequences per step	128	64
Steps	82,970	12,398
Tokens	43.50B	6.50B
Learning rate	warmup 2,000 steps to $3\times 10^{-4}$ , then constant	$3\times 10^{-4}$ to $3\times 10^{-5}$ , linear
Datasets	63	27

Table 4: The two pretraining runs on one node of 8

\times

H100 (Lambda). Wall time is from the node’s own log; tokens per second is the median over steps. The two validation losses are on different held-out sets (each stage holds out the last 0.1% of documents of its own datasets) and are not comparable with each other.

Figure 6: Training loss at every one of the 95,368 logged steps (faint) with an exponential moving average ( $\alpha=0.02$ ), and the learning rate, over the 50B tokens; the inset repeats the curve from 5B tokens on a finer scale. Stable stage: learning rate $3\times 10^{-4}$ , 4K context. Decay: learning rate falling to $3\times 10^{-5}$ , 8K context and a new data mix. The step at the start of decay coincides with the change of data mix and context length, since the learning rate starts falling from its peak only then; the decay losses are measured on different data and do not compare one-to-one with the stable curve.

Figure 7: Stable-stage validation loss against a power law fitted to steps 3K–30K.

Figure 8: Throughput on 8

\times

H100 (median per 500 steps). It stayed flat for the whole run; the decay stage is faster per token at twice the context.

Register (passages)	Step 21,000	End of stable	Final (after decay)	Decay change
Modern formal (143)	1.095	1.051	1.034	$-1.6\%$
Modern colloquial (136)	1.360	1.317	1.298	$-1.4\%$
Tanglish (117)	1.564	1.484	1.417	$-4.5\%$
Literary, OCR (39)	1.287	1.210	1.173	$-3.1\%$
Literary, EPUB (106) a	1.122	1.043	0.909	$-12.8\%$

Table 5: Held-out Tamil, bits per character (lower is better). a See text: these passages are unseen, but other passages of the same books are in the training pack.

Figure 9: Held-out Tamil by register across training (541 held-out passages; each overlaps the training data by less than 5%). Decay steps are counted after the 82,970 stable steps. *EPUB passages are unseen, but other passages of the same books are in the training pack, and the decay mix upweighted those sources (see text).

Figure 10: Base models on Tamil translation and multiple-choice benchmarks, with the same harness and settings for all models (lm-eval, full test sets); dotted lines mark chance. The previous 1.5B model’s FLORES scores use the first 400 sentences (2026-09-18 run), because its outputs ran past the sentence. Clean-subset chrF is within 1.1 of these scores.

	Agaram 2.5B	2.5B (stable end)	Sarvam-1	Agaram 1.5B	Qwen2.5- 1.5B b	Gemma-3- 1B b
FLORES en $\rightarrow$ ta	55.1	49.0	49.2	43.5 a	18.1 a	45.0 a
FLORES ta $\rightarrow$ en	55.5	52.6	44.2	21.2 a	22.7 a	45.8 a
IN22-Conv en $\rightarrow$ ta	40.8	38.6	37.4	31.7	–	–
IN22-Conv ta $\rightarrow$ en	46.0	43.4	37.9	16.5	–	–
IN22-Gen en $\rightarrow$ ta	48.8	44.0	45.9	38.5	–	–
IN22-Gen ta $\rightarrow$ en	54.3	50.1	42.7	19.8	–	–

Table 6: Base-model benchmarks. Translation: chrF, 5-shot. MCQ: accuracy (%); chance is 25 except XCOPA (50) and MultiBLiMP (50). a First 400 FLORES sentences (2026-09-18 run). b Instruction-tuned models, evaluated like the base models. – = not run.

	Base	Epoch 1	Epoch 2	Epoch 3	Mid-run SFT (55K)	Agaram 1.5B
Judged usable (of 100)	–	52	48	45	–	2
multi-turn usable (of 15)	–	6	4	5	–	0
Format checks, set 1	6%	94%	94%	95%	79%	–
Format checks, set 2	9%	88%	89%	89%	68%	–
Stops on its own, set 1	15%	99%	100%	100%	91%	–
Tool call: right tool	0%	96%	96%	96%	56%	–

Table 7: The fine-tuned model by epoch, against the base model, the mid-run SFT on the step-55K base and the \tamilfont அகரம் / Agaram 1.5B release model. Judged usable: first open-chat set, 100 prompts (15 multi-turn). Format checks: share of replies passing all scripted checks. Tool calls: 25 prompts of the second set. IndicIFEval: prompt-level strict accuracy on the native-Tamil subset (430) and the translated subset (489); see Section 7 for the translated subset.

Stage	Areas (new rows)	New rows	Replayed	Epochs run	Kept
General SFT	Section 3.4	134,428	–	3	epoch 3
Round 2	identity, tool results, crisis, tool-call corrections, constraint memory, long answers, false premises, harmful requests, multi-turn repair, verified Thirukkural, format instructions, the model’s charter	7,565	12,000	3	epoch 2
Round 3	identity in English and Tanglish and under pressure, harmful requests, crisis (number first), script switching between turns	2,210	6,000	2	epoch 2
Round 4	identity (false sizes, training data, “are you human”), harmful requests, crisis (secrecy, fear of judgement)	1,451	3,000	2	epoch 1 (released)

Table 8: The SFT lineage of the released model. Each round starts from the checkpoint kept in the row above it.

Set (side)	Items	Clean	Over 50%	What the overlap is
FLORES test, Tamil a	2,024	2,010	0	–
FLORES test, English a	2,024	1,780	146	Wikipedia source sentences
FLORES test, clean pairs	1,012	887	–	per direction
IN22-Conv, Tamil	1,503	1,502	0	–
IN22-Conv, English	1,503	1,493	1	–
IN22-Gen, Tamil	1,024	1,010	0	–

Table 9: Overlap of evaluation sets with the training pack (13-token windows). “Clean” = less than 5% of an item’s windows occur in the pack.

Direction	Pairs (full / clean)	\tamilfont அகரம் / Agaram full	\tamilfont அகரம் / Agaram clean	Sarvam-1 full	Sarvam-1 clean
FLORES en $\rightarrow$ ta	1,012 / 887	55.15	54.97	49.21	49.00
FLORES ta $\rightarrow$ en	1,012 / 887	55.53	55.46	44.23	44.10
IN22-Conv en $\rightarrow$ ta	1,503 / 1,493	40.78	40.71	37.43	37.35
IN22-Conv ta $\rightarrow$ en	1,503 / 1,493	46.02	45.89	37.86	37.82
IN22-Gen en $\rightarrow$ ta	1,024 / 783	48.82	48.69	45.86	45.74
IN22-Gen ta $\rightarrow$ en	1,024 / 783	54.30	53.16	42.73	41.89

Table 10: Translation chrF on the full test sets and on the clean pairs only (both sides under 5% overlap), for the \tamilfont அகரம் / Agaram 2.5B base model and Sarvam-1 (both base models, 5-shot, same harness as Table 6 ).

	Released (chat)	\tamilfont அகரம் / Agaram 2.5B base	Sarvam-1
FLORES en $\rightarrow$ ta / ta $\rightarrow$ en (chrF)	54.3 / 55.2	55.1 / 55.5	49.2 / 44.2
IN22-Conv en $\rightarrow$ ta / ta $\rightarrow$ en	41.0 / 45.9	40.8 / 46.0	37.4 / 37.9
IN22-Gen en $\rightarrow$ ta / ta $\rightarrow$ en	48.4 / 53.9	48.8 / 54.3	45.9 / 42.7
MILU-ta (5-shot)	29.2	29.6	28.8
Belebele-ta	27.3	27.8	58.3
INCLUDE-ta	26.4	26.9	35.5

Table 11: The released chat model against its own base model and Sarvam-1 (a base model), same harness and settings as Table 6 . IndicIFEval: prompt-level strict / loose accuracy on the Tamil subsets; Sarvam-1 was not run.

Job	Hardware	Hours	Cost (USD)
Pretraining, both stages	8 $\times$ H100 SXM5, Lambda	$\approx$ 149 (147.7 training)	$\approx$ 4,765
Abandoned SFT attempt	1 $\times$ H100 PCIe, Lambda (thermally throttled)	3.7	$\approx$ 12
General SFT stage	1 $\times$ H200, JarvisLabs	9.5 training	JarvisLabs credits
SFT rounds 2–4	one DGX Spark (owned)	$\approx$ 12, 4 and 2	–
Teacher generation, evaluation	two DGX Spark machines (owned)	–	–
LLM judge	Kimi K3 through the Fireworks API	–	own funds

Table 12: Compute for \tamilfont அகரம் / Agaram 2.5B. The two Lambda jobs together were invoiced at USD 4,776.50; the split between them is at list price.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 498 papers

Aug 30, 2026

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens, likely aided by a match between our educational-web-text pretraining data and ARC's grade-school-science format rather than a general capability advantage. We report full evaluation results under standard protocols, including a correction to an earlier partial-sample estimate, and findings specific to evaluating small models in a low-resource language: the standard multiple-choice-letter prompt format used by common evaluation harnesses places this model at chance on Nepali reading comprehension, and simultaneously at chance on English in the same format, which would lead a naive benchmark run to conclude the model has no Nepali ability when in fact it does. Concretely, both languages score at chance in the letter-choice format (0.240 Nepali, 0.236 English, against a chance baseline of 0.250), while scoring the answer text directly reveals genuine, English-favoring comprehension (0.306 Nepali, 0.387 English). We describe a manifest-conditioned tool-use contract introduced during instruction tuning, where tool calls are permitted only when a tool manifest is declared in context and suppressed otherwise, and report where that contract holds and where it does not. We release both the base and instruction-tuned model weights under Apache-2.0. The training code and a small privately-sourced portion of the Nepali pretraining corpus are not released; everything needed to reproduce the reported numbers from the released weights is included here.

Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun

Karela Technologies Inc.

Language Model Pretraining Multilingual Language Models Multilingual Language Model Evaluation Low-Resource Language Processing

Ask PDF

Feb 17, 2026

Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language

Can large language models converse in languages virtually absent from their training data? We investigate this question through a case study on Tulu, a Dravidian language with over 2 million speakers but minimal digital presence. Rather than fine-tuning an LLM, we examine whether structured prompts alone can elicit basic conversational ability under controlled prompting. We systematically tackle various challenges posed by absence of training data for Tulu by combining explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play. Evaluated on a manually curated held-out set across three LLMs (Gemini 2.0 Flash, GPT-4o, Llama 3.1 70B) and validated by native speakers, our approach reduces vocabulary contamination from 80% to 5% while achieving 85% grammatical accuracy. Cross-model analysis reveals that negative constraints provide consistent improvements (12--18 percentage points), while grammar documentation effects vary by model architecture (8--22 points).

Prathamesh Devadiga, Paras Chopra

Lossfunk

Low-Resource Language Processing Structured Prompting

Ask PDF

Sep 3, 2026

From Zero to Hero: An Open LLM Ecosystem for Armenian

Pretraining data for Armenian, a morphologically rich and low-resource language, is scarce, and no open Armenian LLM has been released with the data and recipe needed to reproduce it. To address this gap, we curate and release two datasets. ArmWeb is an extensively validated corpus of 4.37M Armenian news documents. ArmSTEM is a parallel English-Armenian collection of 373K math and science problems with step-by-step solutions, translated into Armenian and verified through both answer-preserving LLM judgment and human evaluation. Continued pretraining of Gemma-4-E4B on these datasets yields arm-gemma-e4b, which outperforms every existing open Armenian model as well as its unadapted base, and is the first open Armenian LLM with complete training data and recipe. Our ablations show that news-only continued pretraining improves fluency while eroding knowledge, a pattern we also observe in existing Armenian models, and that a small share of verified translated STEM data reverses the loss. We further find that the largest public Armenian corpora overlap web-derived evaluation panels heavily, including a train/test self-overlap inside FineWeb-2. We openly release all data, models, and code.

Erik Arakelyan, Khatun Avetisyan, Meri Davtyan +5

1NVIDIA · 2COPA

Language Model Pretraining Language Modeling Low-Resource Language Processing

Ask PDF

Abstract

Language Modeling

Explore similar work

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language

From Zero to Hero: An Open LLM Ecosystem for Armenian