PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

Hanif Rahman

cs.CLMar 17, 2026

PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

Authors: Hanif Rahman

Organizations: Independent Researcher

Abstract

We present PashtoCorp, a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains severely underrepresented in NLP. The corpus is assembled from 39 sources spanning seven HuggingFace datasets and 32 purpose-built web scrapers, processed through a reproducible pipeline with Arabic-script tokenization, SHA-256 deduplication, and quality filtering. At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus. Continued MLM pretraining of XLM-R-base on PashtoCorp reduces held-out perplexity by 25.1% (8.08->6.06). On WikiANN Pashto NER, the pretrained model improves entity F1 by 10% relative (19.0%->21.0%) and reduces training variance nearly 7x; the largest gain appears at 50 training sentences (+27%), with PashtoCorp covering 97.9% of WikiANN entity vocabulary. On Belebele Pashto reading comprehension, Gemma-3n achieves 64.6% accuracy, the first published LLM baseline for Pashto on this benchmark. A leave-one-out source ablation shows that Wikipedia (0.7% of documents) is the most critical source for NER: removing it alone reduces entity F1 by 47%. Corpus data, trained model, and code are available at https://huggingface.co/datasets/ihanif/pashto-corpus, https://huggingface.co/ihanif/xlmr-pashto, and https://github.com/ihanif/pashto-corpus.

Figures & tables

Category	Sources	Docs	Words (M)
Web crawl (HuggingFace)	4	1,447,599	773.1
News radio (scraped)	7	1,000,213	222.6
Afghan news (scraped)	12	158,644	77.9
Aggregator / blog	10	164,808	74.5
PDF books / reports	1	16,867	81.6
Encyclopedia	1	19,523	13.6

Table 1: PashtoCorp sources by category.

Filter stage	Docs removed	% of raw
Pashto ratio $<$ 0.70	1,084,231	18.6%
SHA-256 duplicate	356,943	6.1%
Min tokens $<$ 10	42,817	0.7%
Total rejected	1,483,991	25.5%
Retained	2,810,913	74.5%

Table 2: Pipeline rejection statistics.

Corpus	Words	Docs	vs. PashtoCorp
NLPashto [ 6 ]	$\sim$ 15M	—	83 $\times$ smaller
OSCAR 2301	$\sim$ 31M	—	40 $\times$ smaller
CC-100 (ps)	$\sim$ 54M	257K	23 $\times$ smaller
MADLAD-400 (ps)	$<$ 1M	4K	$>$ 1000 $\times$ smaller
GlotCC (pbt-Arab)	—	37K	—
HPLT v2.0 (pbt-Arab)	—	466K	—

Table 3: Comparison with prior Pashto corpora.

Figure 1: Document distribution by domain category ( $N=2{,}810{,}913$ ).

Figure 2: Cumulative unique vocabulary (millions) as sources are incorporated in descending document-count order. Despite its small size (16.9K documents, position 17), FinePDFs contributes the largest marginal vocabulary of any single source: 2.22M unique types.

Group removed	Docs	Vocab	Vocab lost	Entity cov.
— (Full corpus)	—	6,322,778	—	99.5%
PDF Books	16,867	4,103,573	$-$ 35.1%	99.5%
Web Crawl	1,447,599	4,418,444	$-$ 30.1%	98.5%
Afghan News	126,867	6,132,742	$-$ 3.0%	99.5%
Wikipedia	19,523	6,239,562	$-$ 1.3%	99.5%
News Radio	1,060,498	6,267,544	$-$ 0.9%	99.5%

Table 4: Vocabulary ablation (leave-one-out). Entity coverage = WikiANN tokens covered.

Source	Docs	Marginal vocab	% corpus
finepdfs_ps	16,867	2,219,205	35.1%
hplt_ps	397,273	495,782	7.8%
fineweb2_ps	483,890	345,935	5.5%
culturax_ps	335,609	207,160	3.3%
benawa	58,319	201,093	3.2%
tolafghan	20,525	106,514	1.7%

Table 5: Per-source marginal vocabulary (top 10 sources).

Group removed	Docs	PPL	NER F1
Full (750 steps)	2,810,913	6.055	19.6%
Web Crawl	1,363,314	6.581	18.9%
Wikipedia	2,791,390	6.412	10.3%
SWN	2,798,367	6.399	13.6%
PDF Books	2,794,046	6.384	12.9%
Afghan News	2,684,046	6.314	21.5%

Table 6: Pretraining ablation (400 steps each, sorted by PPL). Full 750-step model shown as reference.

Figure 3: Log-log frequency vs. rank for the top $10^{5}$ word types. The Zipf fit ( $\alpha=1.624$ , $R^{2}=0.985$ ) holds closely.

Token set	Covered	Total	Coverage
All WikiANN tokens	2,062	2,151	95.9%
Entity tokens (all)	381	389	97.9%
PER (persons)	164	164	100.0%
LOC (locations)	101	102	99.0%
ORG (organisations)	133	140	95.0%

Table 7: PashtoCorp coverage of WikiANN tokens.

Model	Acc.	Macro-F1
CNN [ 1 ]	92.4%	91.8%
XLM-R-base (published)	94.8%	94.0%
Pashto BERT (published)	94.8%	94.3%
XLM-R-base (our repro)	94.6%	94.1%
XLM-R+PashtoCorp	94.5%	94.0%

Table 8: POLD results (macro-F1).

Model	F1 mean	F1 std	$\Delta$
XLM-R-base	19.0%	$\pm$ 4.7%	—
XLM-R+PashtoCorp	21.0%	$\pm$ 0.7%	+10.3%

Table 9: WikiANN Pashto NER (5 seeds, entity-level F1).

Model	Overall	PER	ORG	LOC
XLM-R-base	13.0% $\pm$ 8.5%	21.8% $\pm$ 15.6%	4.0% $\pm$ 5.7%	9.6% $\pm$ 9.1%
XLM-R+PashtoCorp	20.6% $\pm$ 3.9%	33.1% $\pm$ 5.8%	8.8% $\pm$ 12.4%	10.9% $\pm$ 8.8%

Table 10: NER per-entity-type F1 (3 seeds, mean

\pm

std).

$n$	XLM-R-base	XLM-R+PashtoCorp	$\Delta$
10	8.5% $\pm$ 5.6%	8.1% $\pm$ 5.1%	$-$ 0.4pp
25	10.9% $\pm$ 5.9%	9.3% $\pm$ 4.8%	$-$ 1.6pp
50	18.8% $\pm$ 4.7%	23.8% $\pm$ 7.6%	+5.0pp (+27%)
75	36.1% $\pm$ 6.5%	28.2% $\pm$ 8.0%	$-$ 7.9pp
100	43.4% $\pm$ 12.9%	39.4% $\pm$ 4.2%	$-$ 4.0pp, $3\times$ lower $\sigma$

Table 11: Sample efficiency: entity-level F1 by training size (5 seeds, mean

\pm

std).

Figure 4: WikiANN NER entity F1 vs. training set size (5 seeds, $\pm$ 1 std shaded). PashtoCorp pretraining peaks at $n=50$ (+27% relative F1).

Method	Accuracy
Random baseline	25.0%
MPNet embedding similarity	25.4%
Pashto BERT embedding similarity	24.2%
Qwen3-0.6B (Q8_0, llama-cpp)	27.7%
Qwen3-1.7B (Q8_0, llama-cpp)	33.4%
Qwen3-4B (Q4_K_M, llama-cpp)	37.9%

Table 12: Belebele Pashto (zero-shot, 4-option MC).

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 548 papers

Pashto Common Voice: Building the First Open Speech Corpus for a 60-Million-Speaker Low-Resource Language
Mar 27, 2026Hanif Rahman, Shafeeq ur RehmanSpeech Corpus Construction Low-Resource Language Processing
Pashto DAO
Fine-tuning Whisper for Pashto ASR: strategies and scale
Apr 7, 2026Hanif RahmanAutomatic Speech Recognition Low-Resource Speech Recognition
Independent Researcher.
Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation
Apr 6, 2026Hanif RahmanAutomatic Speech Recognition ASR Evaluation
Independent Researcher

Mar 27, 2026J/K move · Enter open · S save

Pashto Common Voice: Building the First Open Speech Corpus for a 60-Million-Speaker Low-Resource Language

Hanif Rahman, Shafeeq ur Rehman

Pashto DAO

Speech Corpus Construction Low-Resource Language Processing Speech Processing Low-Resource Speech Recognition

Open paper AskPDF arXiv

We present the Pashto Common Voice corpus -- the first large-scale, openly licensed speech resource for Pashto, a language with over 60 million native speakers largely absent from open speech technology. Through a community effort spanning 2022-2025, the corpus grew from 1.5 hours and 5 contributors to 147 total hours and 1,483 unique speakers across ten Mozilla Common Voice releases (CV14-CV23). Speaker participation increased approximately 108-fold between CV17 and CV18, coinciding with a VOA Pashto broadcast campaign. We describe the full methodology: interface localisation, Wikipedia-based sentence extraction with automated filtering, phonemically targeted contributions for the four most frequently dropped Pashto characters, and multi-channel community outreach. MCV23 contains 107,781 clips (60,337 validated; 82.33 validated hours) across 13 content domains. Fine-tuning Whisper Base on the MCV20 yields 13.4% WER on the MCV20 test split, against the published Whisper Base zero-shot WER of 99.0% on Pashto.

Abstract

Low-Resource Language Processing

Explore similar work

Pashto Common Voice: Building the First Open Speech Corpus for a 60-Million-Speaker Low-Resource Language

Fine-tuning Whisper for Pashto ASR: strategies and scale

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation