cs.CLMar 31, 2026

SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali

Authors: Ranidu Gurusinghe, Nevidu Jayatilleke

Organizations: School of Computing, Informatics Institute of Technology, Sri Lanka · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words, incorporating 16 copyright-cleared historical Buddhist documents alongside the complete web-scraped Tripitaka canonical texts. The corpus was created through high-quality OCR using Google Document AI on historical manuscripts, combined with systematic web scraping of canonical repositories, followed by rigorous quality control and metadata annotation. The corpus is organised into language-specific subcorpora: Sinhala and Mixed Sinhala-Pali. We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus. This analysis shows that proprietary models significantly outperform open-source alternatives by factors of three to six times. This corpus supports the pretraining of domain-adapted language models, facilitates historical language analysis, and aids in the development of information retrieval systems for Buddhist scholarship while preserving Sinhala cultural heritage.

Figures & tables

CorpusLanguage(s)TraditionTokensUnitsAnnotationAlignment
Bible 100 ( 2015 )100 langsChristianvaries∼\sim 31K verses/langNoneVerse
CBETA ( 2002 )Classical ChineseBuddhist100M+ chars5,320+ textsTEI XML-
GRETIL ( 2020 )10+ IndicMultiGBs (raw)1,000+ textsRaw text-
Quranic Arabic ( 2010 )ArabicIslamic77,430 words6,236 āyahsMorph+Syn+SemVerse
Tanzil ( 2007 )40+ langsIslamicvaries6,236 verses/langNoneVerse
DCS ( 2020 )SanskritMulti2.5M+ items∼\sim 650K sentsLemma+POS+Morph-
Table 1: Comparison of SiPaKosa with related religious and Indic text corpora.
Figure 1: Complete methodology pipeline showing dual-source data collection, processing, and integration.
Figure 2: Book-level filtering from 83 downloaded books to 16 corpus-eligible books.
SourceMetricCount
Historical PDF CorpusDocuments (books)16
Content pages7,064
Sentences247,513
Web-Scraped Canonical TextsNikayas covered5
Sentences539,326
TotalSentences786,344
Table 2: Overall corpus statistics combining historical and canonical texts.
TypeMetricSinhalaMixed
SentencesPDF corpus111,632135,386
Web-scraped353,907185,419
Total Count465,539320,805
% of corpus59.2%40.8%
TokensTotal Count5,418,7163,831,076
% of corpus58.6%41.4%
Table 3: Sentence and token distribution by language subcorpus and source.
Figure 3: Sentence distribution by language and source.
ModelBuddhistMixedGeneralBS/GM/G
SinhalaSi-PaSinhalaRatioRatio
GPT-3.5-Turbo1.091.111.081.011.03
GPT-4o-mini1.231.151.310.940.88
GPT-4-Turbo1.561.591.481.051.07
GPT-4o1.621.771.680.961.06
Llama-3.1-8B-Instruct3.294.183.261.011.28
Table 4: Comparison of perplexity among proprietary language models and small language models using different corpora.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: PDF processing pipeline applied to the 16 copyright-cleared historical Buddhist books, combining pdfplumber text extraction and Google Document AI OCR , with copyright checking, page classification, and confidence-based quality filtering (threshold: 0.7) prior to corpus integration.
Figure 5: JSON structure of a metadata record of the books in the IFBC corpus, defining the fields and their corresponding data types.
No.TitleAuthorPublished
Books Related to Tipitaka (5 books)
1Maithree Buddha Wanshaya nohoth anagatha wanshayaVen. Wilgammula Sangharaja Thero2015
2Milind PrashnaVen. Heenatikubure Sumangala Nahimi2015
3Pretha WasthuDhammapala Thero2015
4Vimana WasthuDhammapala Thero2015
5Wishuddhi MargayaBuddhaghosa Maha Thero2015
Appendix
Table 5: Copyright-cleared historical Buddhist texts included in the corpus (n=16). All books are in the public domain, with authors who passed away before 1954 (70+ years post-mortem).
More in this topic

Corpus Linguistics

See all 15 papers

Explore similar work

CardsList

  1. SinhaLegal: A Benchmark Corpus for Information Extraction and Analysis in Sinhala Legislative Texts

    Mar 5, 2026Minduli Lasandi, Nevidu JayatillekeLegal NLPLegal Document Analysis

    School of Computing, Informatics Institute of Technology, Sri Lanka · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka

  2. SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0

    Mar 11, 2026Nevidu Jayatilleke, Nisansa de Silva, Uthpala Nimanthi +3

    Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Research Department, Informatics Institute of Technology, Sri Lanka

  3. SinLlama -- A Large Language Model for Sinhala

    Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model PretrainingMultilingual Language Models

    Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1