Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

Toheeb Ogunade

cs.CLAug 19, 2026

Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

Authors: Toheeb Ogunade

Organizations: University of Lagos

Abstract

We ask whether internal representation statistics can provide useful example-level difficulty signals for adaptive inference in multilingual African NLP, and find that they cannot in this setting. Studying natural language inference across 15 African languages with frozen off-the-shelf checkpoints, we report four results. First, AfriXNLI's English configuration shares 1,047 of its 1,050 examples verbatim with XNLI evaluation data, and one widely used NLI checkpoint scores 1.000 on that test split, consistent with XNLI test exposure. Because AfriXNLI is derived from XNLI, its English, French and Swahili configurations cannot serve as clean evaluations for XNLI-trained models. Second, parameter count does not reliably order capability across African languages: our larger checkpoint is better in seven languages and worse in eight, with no significant aggregate difference. Third, across three multilingual representation spaces, angular dispersion is consistently more language-determined than effective rank, so pooled correlations can inflate one and mask the other. Fourth, the association that survives language control depends on the target: effective rank predicts probability gain from escalation but not whether escalation changes the prediction, while cheap-model confidence shows the opposite pattern; the two targets correlate at only 0.655. Under the tested models, signals, and compute budgets, no evaluated signal makes adaptive routing preferable to always-expensive inference, although an oracle exceeds it by 11 accuracy points at 60% of the compute. Our central methodological finding is that a representation statistic can be statistically significant for one notion of computational benefit while being irrelevant to another, and therefore be a poor decision variable.

Figures & tables

Role	Checkpoint	Layers	Hidden	Params
Cheap rung	multilingual-MiniLMv2-L6-mnli-xnli	6	384	$\sim$ 118M
Mid rung	mDeBERTa-v3-base-xnli-…-2mil7	12	768	$\sim$ 278M
Large rung	xlm-roberta-large-xnli	24	1024	$\sim$ 560M
Features (task-tuned)	mDeBERTa-v3-base	12	768	—
Features (African MLM)	afro-xlmr-base	12	768	—
Features (generic MLM)	xlm-roberta-base	12	768	—

Table 1: Frozen checkpoints ( Wang et al., 2021 ; He et al., 2023 ; Conneau et al., 2020 ; Alabi et al., 2022 ) ; no fine-tuning was performed. The three feature sources supply representation statistics only and are never asked to predict. xlm-roberta-base was added post hoc as a replication source (§ 6 ).

§	Result	Depends on a target
4	Benchmark contamination	no
5	Capability not ordered by size	no
6	Language dominates geometry ( $\eta^{2}$ )	no
7	Residual association with either target	yes
8	Routing evaluation	yes

Table 2: Sections marked “no” are computed from accuracies, string matching, and representations alone. A reader who rejects our target construction may discard § 7 –§ 8 without affecting the rest.

	eng		fra		swa
Model	dev	test	dev	test	dev	test
MiniLM-L6	0.887	0.760	0.927	0.720	0.891	0.622
mDeBERTa-base	1.000	0.890	0.998	0.867	0.993	0.742
XLM-R-large	0.998	1.000	0.987	0.995	0.973	0.978

Table 3: Accuracy on the contaminated configurations, full splits (

n=450

dev,

n=600

test). The two models whose cards report training on XNLI development data lose 11–27 points from dev to test. XLM-R-large loses nothing.

Model	Params	Layers	Accuracy	Above chance
MiniLMv2-L6	$\sim$ 118M	6	0.410	11/15
mDeBERTa-v3-base	$\sim$ 278M	12	0.545	14/15
XLM-R-large	$\sim$ 560M	24	0.523	14/15

Table 4: Accuracy on the 15 clean languages (

n=9{,}000

). Chance is

0.333

; “above chance” counts languages whose Wilson 95% lower bound exceeds it.

	mDeBERTa-base	AfroXLMR-base	XLM-R-base
Statistic	(task-tuned)	(African MLM)	(generic MLM)
Angular dispersion	0.583	0.484	0.542
Spectral concentration	0.112	0.249	0.241
Effective rank	0.075	0.133	0.151

Table 5: Between-language variance share

\eta^{2}

, averaged over layers

\{4,8,12\}

. The ordering replicates in all three representation spaces: a task-tuned model, an African-adapted MLM, and a generic multilingual MLM.

Figure 1: Between-language variance share

\eta^{2}

for each geometry statistic in each representation space, averaged over layers

\{4,8,12\}

; the values are those of Table 5 . The ordering—angular dispersion most language-determined, effective rank least—is identical in all three spaces, although the magnitudes differ.

Feature (mDeBERTa)	$\eta^{2}$	Pooled $\rho$	Within $\rho$
L8 angular dispersion	0.757	$+0.296$	$+0.061$
L12 effective rank	0.040	$-0.009$	$-0.127$
L12 spectral concentration	0.031	$+0.044$	$+0.099$

Table 6: Partial Spearman with

\Delta_{\mathrm{prob}}

, controlling for cheap-model confidence, token count, and fragmentation. Angular dispersion is inflated

4.9\times

by pooling; effective rank is masked roughly tenfold.

Figure 2: Pooled versus within-language partial Spearman correlation with

\Delta_{\mathrm{prob}}

, for the statistics of Table 6 . Solid lines mark associations inflated by pooling, dashed lines those masked by it. Angular dispersion, the most language-determined statistic (

\eta^{2}=0.757

), loses most of its apparent relationship once language is controlled; effective rank, the least (

\eta^{2}=0.040

), acquires one.

	$\Delta_{\mathrm{prob}}$		$\Delta_{\mathrm{correct}}$
Signal	$\rho$	Holm $p$	$\rho$	Holm $p$
mDeBERTa L12 effective rank	$\mathbf{-0.127}$	$5.8\!\times\!10^{-32}$	$-0.027$	$0.28$
mDeBERTa L12 spectral conc.	$+0.099$	$2.9\!\times\!10^{-19}$	$+0.003$	$1.00$
mDeBERTa L8 angular dispersion	$+0.061$	$2.2\!\times\!10^{-7}$	$+0.007$	$1.00$
mDeBERTa L12 angular dispersion	$+0.029$	$0.13$	$\mathbf{-0.066}$	$1.6\!\times\!10^{-8}$
Cheap-model confidence	$+0.003$	—	$\mathbf{-0.099}$	—

Table 7: Within-language partial Spearman against both targets, identical procedure, Holm-corrected across all 27 features. Bold marks associations significant at the Holm-adjusted level. The two targets are predicted by different signals.

Budget	Random	Conf.	Geom. $(\Delta_{\mathrm{prob}})$	Geom. $(\Delta_{\mathrm{correct}})$	Geom.+Conf.	Oracle
20%	0.457	0.466	0.461	0.468	0.472	0.613
40%	0.486	0.506	0.498	0.504	0.508	0.688
60%	0.519	0.543	0.524	0.535	0.545	0.688
80%	0.548	0.564	0.553	0.559	0.565	0.688

Table 8: Leave-one-language-out routing accuracy at matched compute budget, 11 viable ladder languages. Always-cheap inference gives

0.426

and always-expensive

0.577

. Training the geometry router on

\Delta_{\mathrm{correct}}

rather than

\Delta_{\mathrm{prob}}

recovers most of its apparent deficit against confidence.

Figure 3: Accuracy–compute frontier over the 11 viable ladder languages, leave-one-language-out; the values are those of Table 8 . The shaded band marks accuracy unreachable by any practical method tested: every routing signal lies below always-expensive inference at every budget, while the oracle exceeds it by 11 points at 60% of the compute.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Lang	$\delta$	Lang	$\delta$	Lang	$\delta$
ibo	$+0.220$	lug	$+0.033$	zul	$-0.003$
sna	$+0.200$	twi	$+0.015$	yor	$-0.010$
sot	$+0.197$	lin	$+0.008$	ewe	$-0.037$
kin	$+0.147$	wol	$-0.002$	hau	$-0.080$
				xho	$-0.087$
				orm	$-0.127$

Appendix

Table 9: Accuracy difference

\delta=

mDeBERTa

-

XLM-R-large per language.

Stability test	Result
Bootstrap, 300 $\times$ within-language resample	$-0.127$ , 95% $[-0.142,-0.111]$
Subsample 25%, 40 seeds	$-0.122\pm 0.019$
Subsample 50%, 40 seeds	$-0.129\pm 0.011$
Leave-one-language-out	$[-0.138,-0.117]$
Label permutation, 20 $\times$	$-0.001$ , $\max\|\rho\|=0.026$

Appendix

Table 10: All resampling variants remain negative; the permutation null is centred at zero with maximum magnitude five times smaller than the effect.

Explore similar work

Jun 2, 2026

Sample-Size Scaling of the African Languages NLI Evaluation

African languages have very little labelled data, and it is unclear if augmenting the quantity of annotation data reliably enhances downstream performance. The study is a systematic sample-size scaling study of natural language inference (NLI) on 16 African languages based on the AfriXNLI benchmark. Under controlled conditions, two multilingual transformer models with roughly 0.6B parameters XLM-R Large fine-tuned on XNLI and AfroXLM-R Large are tested on sample sizes of between 50 and 500 labeled examples and average their results across random subsampling runs. As opposed to the usual belief of monotonic increase with increased data, we find a strongly language sensitive and often non-monotonic scaling behavior. Some languages show early saturation or decrease in performance with sample size as well as high variance in low resource regimes. These results indicate that the volume of data is not enough to guarantee stable profits to African NLI, creating the necessity of language sensitive datasets creation and stronger multi-lingual modelling strategies.

Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion +2

Noida Institute of Engineering and Technology1 · ML Collective1,2,3,4,5

Multilingual Language Model Evaluation Low-Resource Language Processing Natural Language Inference

Ask PDF

Jun 2, 2026

From Script to Semantics: Prompting Strategies for African NLI

Large language models (LLMs) are increasingly evaluated in multilingual settings, yet their inference behavior in low-resource African languages remains underexplored especially under pure prompting without fine-tuning. We present a systematic study of prompting strategies for Natural Language Inference (NLI) in Swahili, Yoruba, and Hausa using the AfriXNLI benchmark. We evaluate five prompting strategies Baseline (zero-shot), Script-Aware, Language Specific, Contrastive, and Native-Label Self-Translation (NL-STP) across two mid-sized open weight models (Llama3.2-3B and Gemma3-4B). To isolate the effect of prompt design, the effect of few-shot examples and Chain-of-Thought reasoning is eliminated in our study. We find a significant difference in performance of class wise across strategies with highly neutral class collapse and high prediction skew in some configurations. Contrastive prompting proves to be the most reliable and steadily improving strategy over language and model and has better balance of class behavior and balance of overall accuracy gains. Notably, well-constructed prompts are sufficient to beat more powerful baselines that are provided with few-shot prompts and Chain-of-Thought prompts. We have found that prompt formulation is essential to multilingual NLI with low-resource languages and that language aware decision structuring can be used to meaningfully enhance robustness in resource challenged settings.

Anuj Tiwari, Terry Oko-odion, Hannah Nwokocha

Noida Institute of Engineering and Technology1, India1 · ML Collective1,2,3, Nigeria2,3

Multilingual Language Model Evaluation LLM Prompting Low-Resource Language Processing Natural Language Inference

Ask PDF

Jan 10, 2026

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present AfriqueLLM, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on Huggingface and Github.

Hao Yu, Tianyi Xu, Michael A. Hedderich +3

McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany +2

Language Model Pretraining Multilingual Language Models Continued Pretraining Low-Resource Language Processing

Ask PDF