Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

Toheeb Ogunade

cs.CLAug 19, 2026

Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

Authors: Toheeb Ogunade

Organizations: University of Lagos

Abstract

We ask whether internal representation statistics can provide useful example-level difficulty signals for adaptive inference in multilingual African NLP, and find that they cannot in this setting. Studying natural language inference across 15 African languages with frozen off-the-shelf checkpoints, we report four results. First, AfriXNLI's English configuration shares 1,047 of its 1,050 examples verbatim with XNLI evaluation data, and one widely used NLI checkpoint scores 1.000 on that test split, consistent with XNLI test exposure. Because AfriXNLI is derived from XNLI, its English, French and Swahili configurations cannot serve as clean evaluations for XNLI-trained models. Second, parameter count does not reliably order capability across African languages: our larger checkpoint is better in seven languages and worse in eight, with no significant aggregate difference. Third, across three multilingual representation spaces, angular dispersion is consistently more language-determined than effective rank, so pooled correlations can inflate one and mask the other. Fourth, the association that survives language control depends on the target: effective rank predicts probability gain from escalation but not whether escalation changes the prediction, while cheap-model confidence shows the opposite pattern; the two targets correlate at only 0.655. Under the tested models, signals, and compute budgets, no evaluated signal makes adaptive routing preferable to always-expensive inference, although an oracle exceeds it by 11 accuracy points at 60% of the compute. Our central methodological finding is that a representation statistic can be statistically significant for one notion of computational benefit while being irrelevant to another, and therefore be a poor decision variable.

Figures & tables

Role	Checkpoint	Layers	Hidden	Params
Cheap rung	multilingual-MiniLMv2-L6-mnli-xnli	6	384	$\sim$ 118M
Mid rung	mDeBERTa-v3-base-xnli-…-2mil7	12	768	$\sim$ 278M
Large rung	xlm-roberta-large-xnli	24	1024	$\sim$ 560M
Features (task-tuned)	mDeBERTa-v3-base	12	768	—
Features (African MLM)	afro-xlmr-base	12	768	—
Features (generic MLM)	xlm-roberta-base	12	768	—

Table 1: Frozen checkpoints ( Wang et al., 2021 ; He et al., 2023 ; Conneau et al., 2020 ; Alabi et al., 2022 ) ; no fine-tuning was performed. The three feature sources supply representation statistics only and are never asked to predict. xlm-roberta-base was added post hoc as a replication source (§ 6 ).

§	Result	Depends on a target
4	Benchmark contamination	no
5	Capability not ordered by size	no
6	Language dominates geometry ( $\eta^{2}$ )	no
7	Residual association with either target	yes
8	Routing evaluation	yes

Table 2: Sections marked “no” are computed from accuracies, string matching, and representations alone. A reader who rejects our target construction may discard § 7 –§ 8 without affecting the rest.

	eng		fra		swa
Model	dev	test	dev	test	dev	test
MiniLM-L6	0.887	0.760	0.927	0.720	0.891	0.622
mDeBERTa-base	1.000	0.890	0.998	0.867	0.993	0.742
XLM-R-large	0.998	1.000	0.987	0.995	0.973	0.978

Table 3: Accuracy on the contaminated configurations, full splits (

n=450

dev,

n=600

test). The two models whose cards report training on XNLI development data lose 11–27 points from dev to test. XLM-R-large loses nothing.

Model	Params	Layers	Accuracy	Above chance
MiniLMv2-L6	$\sim$ 118M	6	0.410	11/15
mDeBERTa-v3-base	$\sim$ 278M	12	0.545	14/15
XLM-R-large	$\sim$ 560M	24	0.523	14/15

Table 4: Accuracy on the 15 clean languages (

n=9{,}000

). Chance is

0.333

; “above chance” counts languages whose Wilson 95% lower bound exceeds it.

	mDeBERTa-base	AfroXLMR-base	XLM-R-base
Statistic	(task-tuned)	(African MLM)	(generic MLM)
Angular dispersion	0.583	0.484	0.542
Spectral concentration	0.112	0.249	0.241
Effective rank	0.075	0.133	0.151

Table 5: Between-language variance share

\eta^{2}

, averaged over layers

\{4,8,12\}

. The ordering replicates in all three representation spaces: a task-tuned model, an African-adapted MLM, and a generic multilingual MLM.

Figure 1: Between-language variance share

\eta^{2}

for each geometry statistic in each representation space, averaged over layers

\{4,8,12\}

; the values are those of Table 5 . The ordering—angular dispersion most language-determined, effective rank least—is identical in all three spaces, although the magnitudes differ.

Feature (mDeBERTa)	$\eta^{2}$	Pooled $\rho$	Within $\rho$
L8 angular dispersion	0.757	$+0.296$	$+0.061$
L12 effective rank	0.040	$-0.009$	$-0.127$
L12 spectral concentration	0.031	$+0.044$	$+0.099$

Table 6: Partial Spearman with

\Delta_{\mathrm{prob}}

, controlling for cheap-model confidence, token count, and fragmentation. Angular dispersion is inflated

4.9\times

by pooling; effective rank is masked roughly tenfold.

Figure 2: Pooled versus within-language partial Spearman correlation with

\Delta_{\mathrm{prob}}

, for the statistics of Table 6 . Solid lines mark associations inflated by pooling, dashed lines those masked by it. Angular dispersion, the most language-determined statistic (

\eta^{2}=0.757

), loses most of its apparent relationship once language is controlled; effective rank, the least (

\eta^{2}=0.040

), acquires one.

	$\Delta_{\mathrm{prob}}$		$\Delta_{\mathrm{correct}}$
Signal	$\rho$	Holm $p$	$\rho$	Holm $p$
mDeBERTa L12 effective rank	$\mathbf{-0.127}$	$5.8\!\times\!10^{-32}$	$-0.027$	$0.28$
mDeBERTa L12 spectral conc.	$+0.099$	$2.9\!\times\!10^{-19}$	$+0.003$	$1.00$
mDeBERTa L8 angular dispersion	$+0.061$	$2.2\!\times\!10^{-7}$	$+0.007$	$1.00$
mDeBERTa L12 angular dispersion	$+0.029$	$0.13$	$\mathbf{-0.066}$	$1.6\!\times\!10^{-8}$
Cheap-model confidence	$+0.003$	—	$\mathbf{-0.099}$	—

Table 7: Within-language partial Spearman against both targets, identical procedure, Holm-corrected across all 27 features. Bold marks associations significant at the Holm-adjusted level. The two targets are predicted by different signals.

Budget	Random	Conf.	Geom. $(\Delta_{\mathrm{prob}})$	Geom. $(\Delta_{\mathrm{correct}})$	Geom.+Conf.	Oracle
20%	0.457	0.466	0.461	0.468	0.472	0.613
40%	0.486	0.506	0.498	0.504	0.508	0.688
60%	0.519	0.543	0.524	0.535	0.545	0.688
80%	0.548	0.564	0.553	0.559	0.565	0.688

Table 8: Leave-one-language-out routing accuracy at matched compute budget, 11 viable ladder languages. Always-cheap inference gives

0.426

and always-expensive

0.577

. Training the geometry router on

\Delta_{\mathrm{correct}}

rather than

\Delta_{\mathrm{prob}}

recovers most of its apparent deficit against confidence.

Figure 3: Accuracy–compute frontier over the 11 viable ladder languages, leave-one-language-out; the values are those of Table 8 . The shaded band marks accuracy unreachable by any practical method tested: every routing signal lies below always-expensive inference at every budget, while the oracle exceeds it by 11 points at 60% of the compute.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Lang	$\delta$	Lang	$\delta$	Lang	$\delta$
ibo	$+0.220$	lug	$+0.033$	zul	$-0.003$
sna	$+0.200$	twi	$+0.015$	yor	$-0.010$
sot	$+0.197$	lin	$+0.008$	ewe	$-0.037$
kin	$+0.147$	wol	$-0.002$	hau	$-0.080$
				xho	$-0.087$
				orm	$-0.127$

Appendix

Table 9: Accuracy difference

\delta=

mDeBERTa

-

XLM-R-large per language.

Stability test	Result
Bootstrap, 300 $\times$ within-language resample	$-0.127$ , 95% $[-0.142,-0.111]$
Subsample 25%, 40 seeds	$-0.122\pm 0.019$
Subsample 50%, 40 seeds	$-0.129\pm 0.011$
Leave-one-language-out	$[-0.138,-0.117]$
Label permutation, 20 $\times$	$-0.001$ , $\max\|\rho\|=0.026$

Appendix

Table 10: All resampling variants remain negative; the permutation null is centred at zero with maximum magnitude five times smaller than the effect.

Explore similar work

Sample-Size Scaling of the African Languages NLI Evaluation
Jun 2, 2026Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion +2Multilingual Language Model Evaluation Low-Resource Language Processing
Noida Institute of Engineering and Technology1 · ML Collective1,2,3,4,5
From Script to Semantics: Prompting Strategies for African NLI
Jun 2, 2026Anuj Tiwari, Terry Oko-odion, Hannah NwokochaMultilingual Language Model Evaluation LLM Prompting
Noida Institute of Engineering and Technology1, India1 · ML Collective1,2,3, Nigeria2,3
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
Jan 10, 2026Hao Yu, Tianyi Xu, Michael A. Hedderich +3Language Model Pretraining Multilingual Language Models
McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany +2

Jun 2, 2026J/K move · Enter open · S save

Sample-Size Scaling of the African Languages NLI Evaluation

Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale+1

Noida Institute of Engineering and Technology1 · ML Collective1,2,3,4,5

Multilingual Language Model Evaluation Low-Resource Language Processing Natural Language Inference

Open paper AskPDF arXiv

African languages have very little labelled data, and it is unclear if augmenting the quantity of annotation data reliably enhances downstream performance. The study is a systematic sample-size scaling study of natural language inference (NLI) on 16 African languages based on the AfriXNLI benchmark. Under controlled conditions, two multilingual transformer models with roughly 0.6B parameters XLM-R Large fine-tuned on XNLI and AfroXLM-R Large are tested on sample sizes of between 50 and 500 labeled examples and average their results across random subsampling runs. As opposed to the usual belief of monotonic increase with increased data, we find a strongly language sensitive and often non-monotonic scaling behavior. Some languages show early saturation or decrease in performance with sample size as well as high variance in low resource regimes. These results indicate that the volume of data is not enough to guarantee stable profits to African NLI, creating the necessity of language sensitive datasets creation and stronger multi-lingual modelling strategies.

Abstract

Explore similar work

Sample-Size Scaling of the African Languages NLI Evaluation

From Script to Semantics: Prompting Strategies for African NLI

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages