TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Ireddi Rakshitha; Devavarapu Yashwanth; Ntakirutimana Pierre

cs.CLAug 27, 2026

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Authors: Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

Organizations: Barclays · Carnegie Mellon University

Abstract

We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedicated tabular representation learning resource. TabuLM extends KinyaBERT-large, a two-tier morphological transformer, with additive row, column, and cell-type embeddings and a learned table-structure attention bias that sharpens same-row and same-column attention. Pre-training uses two new objectives: Masked Cell Recovery (MCR), which masks entire cells and forces reconstruction from row and column context, and Column Type Prediction (CTP), which predicts column semantic types from observed cell values. We pre-train on 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals, and introduce TabQA-kin, the first native Kinyarwanda table question-answering benchmark comprising 526 QA pairs across 31 tables and four question types. TabuLM achieves 62.0% exact match on TabQA-kin, outperforming KinyaBERT-large by 5.7 EM points and all multilingual baselines (mBERT 49.3%, XLM-R 50.0%) by 11.7-12.7 points. Analysis shows that structural table embeddings are most decisive for comparison and lookup questions, while morphological awareness provides complementary gains. Our code, data, and pre-trained checkpoint are publicly available.

Figures & tables

Panel of Figure 1 — Figure 1: Table serialization. A two-row Kinyarwanda agricultural table (a) is linearised into a flat token sequence (b). Red [CLS]/[TAB] open the table; orange [ROW] separates rows; green [CEL] precedes each cell. Grid coordinates $(r,c)$ are attached to every token and feed the row, column, and cell-type embedding matrices as well as the MCR and CTP objectives.

Panel of Figure 2 — Figure 2: TabuLM architecture. Orange components are new relative to KinyaBERT. Three inputs converge at Add & Concatenate : the morphological summary $\mathbf{m}_{w}$ (from Tier 1), the stem embedding $\mathbf{s}_{w}$ , and the tabular embeddings. The table-structure bias is injected inside the Sequence Transformer at every layer.

Panel of Figure 3 — Figure 3: TabuLM pre-training objectives. (a) Masked Cell Recovery (MCR): entire cells are masked (gray); the model reconstructs cell content using same-row context (row identity) and same-column context (distributional signal) via the tabular embeddings and attention bias. (b) Column Type Prediction (CTP): a column header is masked; the model predicts the column’s semantic type ( Numeric , Text , Categorical , or Date ) from the observed cell values.

Panel of Figure 4 — Figure 4: TabuLM data-collection and training pipeline. Starting from Rwandan government open-data portals, tables are extracted, serialized with structural tokens, and used to pre-train TabuLM on the MCR and CTP objectives. The resulting checkpoint is fine-tuned on TabQA-kin to produce the final results.

Resource	Count	Source
Pre-training tables	172	NISR, RAB, REB, MoH
Pre-training cells	$\sim$ 35,000
Pre-training sequences	$\sim$ 170/loop
TabQA-kin total QA pairs	526	Template generation
lookup	186
comparison	123

Table 1: Dataset statistics for TabuLM pre-training corpus and TabQA-kin benchmark.

Hyperparameter	Pre-training	Fine-tuning
Optimizer	LAMB	AdamW
Peak LR	$4\times 10^{-4}$	$2\times 10^{-5}$
LR schedule	Linear warmup + decay	Constant
Warmup steps	500 (5%)
Weight decay	0.01	0.01
Effective batch	64	16

Table 2: TabuLM hyperparameters for pre-training and fine-tuning.

Model	Dev EM
GPT-4o (zero-shot)	64.0
GPT-4o-mini (zero-shot)	64.0
mBERT	49.3
XLM-R	50.0
KinyaBERT-large	56.3
TabuLM (full)	62.0

Table 3: TabQA-kin dev set overall EM. GPT-4o and GPT-4o-mini both score 64.0%, showing the LLM ceiling is not scale-dependent. TabuLM outperforms all fine-tuned baselines by 5.7–12.7 EM points; all fine-tuned models substantially outperform both LLMs on aggregation (see Table 4 ).

Model	Look.	Comp.	Agg.	All
GPT-4o (0-shot)	82.9	79.2	25.9	64.0
GPT-4o-mini (0-shot)	85.7	70.8	29.6	64.0
mBERT	16.7	50.0	80.8	49.3
XLM-R	19.2	44.4	85.2	50.0
KinyaBERT	26.7	59.1	88.9	56.3
TabuLM	28.6	66.7	79.2	62.0

Table 4: EM per question type on TabQA-kin dev set. All fine-tuned models use question-guided joint row–column filtering for lookup and top-2 question-relevant row restriction for comparison . GPT-4o (zero-shot, 86 evaluable items) leads on lookup and comparison but is substantially outperformed on aggregation by all fine-tuned models. Bold = best per column (GPT-4o lookup/comparison excluded from fine-tuned best).

Configuration	Dev EM	$\Delta$
TabuLM (full)	62.0
$-$ Table-structure bias	64.0	$+2.0^{\dagger}$
$-$ Row/col/cell-type embeds	58.0	$-4.0$
$-$ MCR objective	64.0	$+2.0^{\dagger}$
$-$ CTP objective	64.0	$+2.0^{\dagger}$

Table 5: Ablation study (TabQA-kin dev set, EM).

\Delta

= difference from full model. † Variants exceeding full model are within noise (50-item dev set; 1 item = 2% EM). Only the tabular embeddings show a robust effect; bias and CTP components provide no detectable benefit at this pre-training scale.

Question (en gloss)	Gold	TabuLM	KinyaBERT
Between Ibishyimbo and Uburo , which has higher 2021 export growth? ( exports_trade_2022 )	Ibishyimbo	✓	$\times$ (Ibikorei)
Between Ibigori and Ibirayi , which has more farmland (ha)? ( crop_production_2022 )	Ibigori	✓	$\times$ (Amasaka)

Table 6: Comparison questions where TabuLM succeeds and KinyaBERT fails. KinyaBERT ignores the two-entity constraint and selects the global column maximum; TabuLM correctly restricts comparison to the two named rows.

	TabuLM	GPT-4o
Parameters	65M	${\sim}$ 1.8T
Inference	Local GPU	Proprietary API
Per-query cost	$0 (local)	${\sim}$ $0.01
GPU required	RTX 3090 (24 GB)	None (API)
Fine-tunable	Yes	No ∗
Agg. EM	79.2%	25.9%

Table 7: Deployment comparison. ∗ GPT-4o fine-tuning is available via API but was not evaluated here; results are zero-shot.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Kinyarwanda Template	English Gloss
Lookup
Ni angahe [COL] ya [ENT]?	What is [COL] of [ENT]?
[ENT] yari ifite angahe [COL]?	How much [COL] did [ENT] have?
Comparison
Ni [ENT1] cyangwa [ENT2] ifite [COL] nyinshi?	Does [ENT1] or [ENT2] have more [COL]?
Hagati ya [ENT1] na [ENT2], iyihe ifite [COL] nkeya?	Between [ENT1] and [ENT2], which has less [COL]?

Appendix

Table 8: Kinyarwanda question templates for TabQA-kin. [COL] = column header (e.g., ubuhinzi “cultivated area”), [ENT] = row entity from the first column (e.g., Gasabo ), [TYPE] = entity type label (e.g., akarere “district”), [VAL] = numeric threshold. Count questions are excluded from cell-selection evaluation.

Akarere	Ubuhinzi (ha)	Umusaruro (t)	Sezon
(District)	(Cultivated area)	(Yield, metric tons)	(Season)
Gasabo	1,847	4,203	2022A
Kicukiro	923	2,156	2022A
Nyarugenge	412	847	2022A
Bugesera	12,304	38,145	2022A

Appendix

Table 9: Example pre-training table from the Rwanda Agriculture Board. Kinyarwanda column headers are given with English translations. The MCR objective masks entire cells (e.g., Bugesera’s yield 38,145 t) and requires reconstruction from row and column context; CTP predicts the numeric type for the Umusaruro column.

Hyperparameter	Pre-training	Fine-tuning
Optimizer	LAMB	AdamW
Peak learning rate	$4\times 10^{-4}$	$2\times 10^{-5}$
LR schedule	Warmup + decay	Constant
Warmup steps	500	50
Effective batch size	64 (8 $\times$ GA8)	16
Max sequence length	512	512

Appendix

Table 10: Complete hyperparameters for TabuLM pre-training and fine-tuning. All ablation variants use identical fine-tuning hyperparameters.

Question (en gloss)	Type	Gold	TabuLM	Failure mode
Which crop has the most farmland in Kigali? ( crop_land_2022 )	Agg	Ibigori	✓
Which district has least health workers?	Agg	Nyarugenge	$\times$ (Gasabo)	Wrong column alignment
Between Ibigori and Ibishyimbo, which has higher 2023 yield?	Cmp	Ibigori	✓
What is the enrollment of Nyamata SS?	Lkp	1,240	$\times$ (963)	Duplicate values in column
How many students enrolled in Remera?	Lkp	Remera PS	✓

Appendix

Table 11: Additional TabuLM error analysis examples. “Wrong column alignment” occurs when the question’s column cue matches multiple column headers; “Duplicate values” occurs when the same numeric value appears in more than one row, defeating the column-wise disambiguation heuristic.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 418 papers

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training
Aug 27, 2026Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana PierreSentence Embeddings African Language Processing
Barclays · Carnegie Mellon University
TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings
Oct 12, 2026Félix Lefebvre, David Holzmüller, Gioia Blayer +3Tabular ML Tabular Representation Learning
SODA Team, Inria Saclay · SAP SE · Probabl
Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks
Apr 23, 2026Liane Vogel, Kavitha Srinivas, Niharika D'Souza +3Tabular Foundation Models Representation Learning
Technical University of Darmstadt · IBM Research, USA

Aug 27, 2026J/K move · Enter open · S save

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

Barclays · Carnegie Mellon University

Sentence Embeddings African Language Processing Low-Resource Language Processing Contrastive Learning Text Embedding Evaluation

Open paper AskPDF arXiv

We present KinyaEmbed, the first dedicated sentence embedding model for Kinyarwanda, a morphologically rich Bantu language spoken by over 12 million people in Rwanda. Existing multilingual embedding models such as LaBSE, mE5-large, and OpenAI text-embedding-3-large perform poorly on Kinyarwanda due to severe under-representation in their pre-training corpora. KinyaEmbed is built on KinyaBERT-large and trained via a four-stage curriculum using MultipleNegativesRankingLoss (MNRL): Stage 1 leverages ~18,000 paraphrase pairs from the Official Gazette of Rwanda with three temperature scales; Stage 2 fine-tunes on 715 NLLB-translated MNLI triplets for entailment structure; Stage 3 aligns representations using English-Kinyarwanda OPUS-100 translation pairs; Stage 4 refines with 2,936 high-quality pairs filtered from KinyaCOMET at quality threshold 0.8. We evaluate on SemRel2024-rw and introduce Wiki-RW-STS, a new contamination-free Kinyarwanda STS benchmark of 300 pairs derived from Kinyarwanda Wikipedia. A seven-checkpoint ensemble (all5+23A*2, with the final stage double-weighted) achieves Spearman \r{ho}=0.7298 on SemRel2024-rw, surpassing mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%. KinyaEmbed also achieves the best document clustering silhouette score (0.2146) across all evaluated models. All checkpoints, the KinyaCOMET filtered pairs, and the Wiki-RW-STS benchmark are publicly available.

Abstract

Language Model Pretraining

Explore similar work

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks