The Lexical Imprint of Pre-training Data

Jeonghoon Kim; Woojin Chung; Woomin Song; Cheonbok Park; Nako Sung; Jinwoo Shin

cs.CLFeb 11, 2026

The Lexical Imprint of Pre-training Data

Authors: Jeonghoon Kim, Woojin Chung, Woomin Song, Cheonbok Park, Nako Sung, Jinwoo Shin

Organizations: NAVER Cloud · Korea Advanced Institute of Science and Technology (KAIST)

Abstract

How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a target benchmark. Likelihood training shapes the model's predictive distribution from the pre-training corpus, and zero-shot scores under likelihood-based evaluation depend in part on the lexical alignment between that corpus and the benchmark target. Cleanly tracing how the lexical imprint of pre-training data scales with corpus--benchmark alignment requires controlled from-scratch pre-training that varies corpus identity while holding the training recipe fixed. Across this controlled sweep, spanning $10$ zero-shot benchmarks, $4$ pre-training corpora, and $3$ model scales, we find that corpus--benchmark lexical alignment, UCE, consistently tracks zero-shot accuracy on every benchmark, with corpora more aligned with the benchmark (lower UCE) achieving higher accuracy. The same signal extends from measurement to control at finer scopes. At the document level, strengthening the imprint by selecting training documents to maximize lexical alignment with the benchmark produces the accuracy gains predicted by the mechanism. At the prompt level, matching the imprint per prompt by routing each prompt to the drafter whose domain corpus is most lexically aligned with it produces measurable throughput gains over a generalist-drafter baseline. Taken together, pre-training corpora leave a measurable lexical imprint on base-model zero-shot behavior, and UCE provides a model-free probe of lexical alignment between reference and target data.

Figures & tables

Figure 1: Training on a corpus with word distribution

P

leaves a lexical imprint on the model, partly visible in likelihood scores on a target benchmark with word distribution

Q

. Word-level unigram cross-entropy (UCE), a tokenizer-agnostic measure of

P

–

Q

alignment, is a proxy for this imprint. Across training corpora, lower UCE goes with higher zero-shot accuracy on the same benchmark. Selecting training documents that lower UCE strengthens the imprint and raises target accuracy. Routing each prompt to the drafter whose corpus has the lowest UCE against it raises decoding speed.

Figure 2: Word-level unigram cross-entropy (left axis, bars, bits

\downarrow

) and zero-shot accuracy (right axis, diamonds) across

8

benchmarks for a

3.36\mathrm{B}

model trained on

60\mathrm{B}

tokens of C4 , DCLM , or FineWeb-Edu . Lower cross-entropy tracks higher accuracy in every panel. Full per-task results across all scales are in Appendix D.4 .

Figure 3: Cross-entropy (bits

\downarrow

) versus zero-shot accuracy for

N{=}16

random

8.5\mathrm{B}

-token subsets at the

400\mathrm{M}

scale (

5

each from C4 , DCLM , and FineWeb-Edu , and

1

from OpenWebText ). Solid dots mark per-subset points, open diamonds with whiskers mark per-corpus mean

\pm 1\sigma

, and the dashed line is the OLS fit with its

95\%

confidence band.

\Delta H_{i}\leftarrow H(Q,P^{\setminus r_{i}})-H(Q,P)

Algorithm 1 One-shot row selection in the UCE filter

	Targeted ( $7$ -task)				Out-of-target ( $4$ -task)
Source	Full	Random	DSIR [ 43 ]	UCE-Filter	Full	Random	DSIR	UCE-Filter
DCLM	11.17	11.17	11.08	10.92	11.39	11.39	11.43	11.41
FineWeb-Edu	11.32	11.32	11.20	10.93	11.62	11.62	11.65	11.58

Table 1: Corpus–benchmark UCE (bits

\downarrow

) of each source corpus (Full) and of the subsets selected from it. Targeted values are measured against the word distribution pooled over the

7

tasks used for selection (ARC-Easy, ARC-Challenge, HellaSwag, SciQ, OpenBookQA, PIQA, SWAG), and Out-of-target values against that of the

4

other tasks (LAMBADA, CommonsenseQA, BoolQ, SocialIQA). The Out-of-target block is measured on a fixed-seed sample of each subset, with Full repeating Random, and its values are comparable only within the block.

		Targeted ( $7$ -task) avg. (%)			Out-of-target ( $4$ -task) avg. (%)
Source	Model	Random	DSIR [ 43 ]	UCE-Filter	Random	DSIR	UCE-Filter
DCLM	$1.34\mathrm{B}$	58.85 $\pm$ 0.56	59.85 $\pm$ 0.53	62.49 $\pm$ 0.26	46.57 $\pm$ 0.53	43.79 $\pm$ 0.23	46.22 $\pm$ 0.94
DCLM	$3.33\mathrm{B}$	62.95 $\pm$ 0.03	63.01 $\pm$ 0.33	66.26 $\pm$ 0.20	48.75 $\pm$ 0.53	46.23 $\pm$ 0.38	49.56 $\pm$ 0.09
FineWeb-Edu	$1.34\mathrm{B}$	59.12 $\pm$ 0.41	60.15 $\pm$ 0.08	61.85 $\pm$ 0.15	42.60 $\pm$ 1.06	41.14 $\pm$ 1.37	43.40 $\pm$ 0.29
FineWeb-Edu	$3.33\mathrm{B}$	62.80 $\pm$ 0.09	62.77 $\pm$ 0.18	64.64 $\pm$ 0.21	44.85 $\pm$ 0.49	43.39 $\pm$ 0.33	45.61 $\pm$ 0.27

Table 2: Zero-shot accuracy of

1.34\mathrm{B}

and

3.33\mathrm{B}

models trained from scratch on

70\mathrm{B}

tokens of each selected subset, as mean

\pm

standard deviation over three training seeds. Targeted averages the

7

tasks used for selection (ARC-Easy, ARC-Challenge, HellaSwag, SciQ, OpenBookQA, PIQA, SWAG), and Out-of-target the

4

other tasks (LAMBADA, CommonsenseQA, BoolQ, SocialIQA).

Stage	Strategy	Tokens	MMLU	MMLU-Pro
Stage 1	Endpoint ( $=$ Stage 2 init)	init.	71.17	44.46
Stage 2	Production Mix (baseline)	${\sim}2\mathrm{T}$	75.61	48.64
Stage 2	UCE-Filter Web Subset	$\mathbf{200}\mathrm{B}$	74.28	48.81

Table 3: Zero-shot MMLU and MMLU-Pro accuracy of a

30\mathrm{B}

-parameter model after Stage 1 and after Stage 2, where Stage 2 uses either the production mixture or the same mixture with the web component replaced by its UCE-filtered subset. Tokens is the number of Stage 2 training tokens.

	AIME		LiveCodeBench		MMLU		MedQA		Average
	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup
Base Model	1.000	-	1.000	-	1.000	-	1.000	-	1.000	-
Generalist Drafter	1.702	55.6%	1.587	53.0%	1.436	45.7%	1.409	45.7%	1.534	50.0%
UCE-Based Routing	1.707	61.6%	1.615	54.7%	1.451	55.3%	1.426	49.6%	1.550	55.3%

Table 4: Mean acceptance length

\tau

and per-token decoding speedup of speculative decoding with DeepSeek-R1-Distill-Llama-8B as the verifier, using a generalist drafter or UCE-based routing among specialist drafters. Speedup is relative to the verifier alone (Base Model).

	ARC-Challenge	ARC-Easy	BoolQ	HellaSwag	LAMBADA	MMLU	OpenBookQA	PIQA	SciQ	SocialIQA
DCLM	0.08	0.02	3.07	0.11	0.00	0.64	0.00	0.12	0.64	0.00
FineWeb-Edu	0.04	0.06	11.25	0.29	0.00	1.39	0.02	0.12	4.91	0.00
C4	0.00	0.02	3.14	0.12	0.00	0.37	0.02	0.07	0.25	0.01
OpenWebText	0.00	0.02	5.35	0.02	0.06	0.58	0.00	0.02	0.15	0.00

Table 5: Share of benchmark items (%, all splits) with at least one word-level

13

-gram found in a sample of up to one million documents per corpus.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	$\beta_{1}$	$\beta_{2}$	$\epsilon$	Weight Decay
AdamW	0.9	0.999	$10^{-8}$	0.1

Appendix

Table 6: Training configurations for the corpus-ranking experiments (Section 4.2 ). All runs use this recipe across the three model scales of Table 7 .

Size	$n_{\text{layers}}$	$n_{\text{heads}}$	$d_{\text{model}}$	$d_{\text{ffn}}$	$d_{\text{head}}$
402M	24	8	1024	4096	128
1.33B	30	30	1920	5120	64
3.36B	32	20	2560	10240	128

Appendix

Table 7: Model configurations for the corpus-ranking experiments (Section 4.2 ). Sizes count non-embedding parameters.

Optimizer	$\beta_{1}$	$\beta_{2}$	$\epsilon$	Weight Decay
AdamW	0.9	0.95	$10^{-8}$	0.033

Appendix

Table 8: Training configurations for the UCE-filter experiments (Section 5.2 ). All sixteen runs (Source

\times

Scale

\times

{UCE-filter, Random, DSIR-paper, DSIR-LN}) use this recipe.

Size	$n_{\text{layers}}$	$n_{\text{heads}}$	$d_{\text{model}}$	$d_{\text{ffn}}$	$d_{\text{head}}$	Pattern
1.34B	24	16	2048	5632	128	DCLM 1B-1x
3.33B	28	24	3072	8192	128	DCLM 7B (scaled to 28 layers)

Appendix

Table 9: Model architectures for the UCE-filter experiments (Section 5.2 ). DataComp-LM patterns [ 28 ] . All use SwiGLU activations, RMSNorm, RoPE positions, and tied input/output embeddings.

Source	Benchmark	UCE	Type cov.	Mass cov.	$m_{\text{unc}}$	Uncov. contrib.	Share of UCE
DCLM	CSR7	$11.1665$	$95.54\%$	$99.973\%$	$2.73\times 10^{-4}$	$0.0095$	$0.085\%$
DCLM	Corpus-ranking	$11.1843$	$95.83\%$	$99.972\%$	$2.80\times 10^{-4}$	$0.0098$	$0.087\%$
FineWeb-Edu	CSR7	$11.3256$	$93.88\%$	$99.957\%$	$4.32\times 10^{-4}$	$0.0146$	$0.128\%$
FineWeb-Edu	Corpus-ranking	$11.3572$	$94.12\%$	$99.955\%$	$4.50\times 10^{-4}$	$0.0151$	$0.133\%$

Appendix

Table 10: UCE decomposition by coverage (bits

\downarrow

) for DCLM and FineWeb-Edu against the CSR7 and ten-task corpus-ranking benchmark groupings, measured on a five-percent fragment sample of each source corpus. The benchmark mass on uncovered words

m_{\text{unc}}

and its contribution to UCE are computed from Equation ( 8 ).

Variant	DCLM		FineWeb-Edu		$\Delta_{\text{Fw}-\text{DCLM}}$
Variant	CSR7	Corpus-rank	CSR7	Corpus-rank	CSR7	Corpus-rank
Baseline	$11.1650$	$11.1832$	$11.3215$	$11.3539$	$+0.1565$	$+0.1707$
Lowercase	$10.8486$	$10.8652$	$10.9952$	$11.0246$	$+0.1466$	$+0.1594$
Lidstone ( $\alpha=\tfrac{1}{2}$ )	$11.1666$	$11.1848$	$11.3243$	$11.3566$	$+0.1577$	$+0.1718$
No smoothing	$11.1656$	$11.1838$	$11.3234$	$11.3556$	$+0.1578$	$+0.1718$
IDF weighting	$14.3153$	$14.3110$	$14.5502$	$14.5539$	$+0.2349$	$+0.2429$

Appendix

Table 11: UCE (bits

\downarrow

) under five preprocessing variants on the DCLM and FineWeb-Edu corpora (one-percent fragment sample), evaluated against the seven-task targeted grouping (CSR7) of Section 5.2 and the ten-task corpus-ranking grouping of Section 4.2 .

\Delta_{\text{Fw}-\text{DCLM}}

is the within-grouping FineWeb-Edu minus DCLM UCE.

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	83.97% (150,599)	98.15% (176,031)	1.84% (3,298)	99.99% (179,329)	11.1946	73.65
	DCLM	78.54% (140,870)	97.24% (174,411)	2.75% (4,926)	99.99% (179,337)	11.5477	70.20
	C4	77.74% (139,428)	96.63% (173,311)	3.35% (6,003)	99.98% (179,314)	11.6493	60.77
ARC-Challenge	FineWeb-Edu	85.04% (93,417)	98.16% (107,823)	1.82% (1,997)	99.97% (109,820)	11.0831	38.82
	DCLM	80.32% (88,236)	97.45% (107,051)	2.54% (2,791)	99.99% (109,842)	11.3847	34.81
	C4	79.88% (87,745)	97.05% (106,610)	2.89% (3,180)	99.95% (109,790)	11.4714	26.79

Appendix

Table 12: Per-task word-frequency coverage, unigram cross-entropy (bits

\downarrow

), and zero-shot score for a

3.36\mathrm{B}

model on

60\mathrm{B}

tokens. LAMBADA is reported in perplexity (lower better) and other tasks in accuracy (higher better). The

\sim

80% and

\sim

95% columns report cumulative shares of benchmark word mass attributable to top-percentile training words, the 95–100% column the share on the remaining tail words, and Seen is the fraction of benchmark vocabulary observed at least once. Source for Figure 2 .

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score ( $1.33\mathrm{B}$ )	Score ( $402\mathrm{M}$ )
ARC-Easy	FineWeb-Edu	83.87% (150,418)	98.09% (175,920)	1.90% (3,407)	99.98% (179,327)	11.1921	65.87	62.42
	DCLM	78.55% (140,887)	97.24% (174,412)	2.74% (4,923)	99.99% (179,335)	11.5461	63.17	56.78
	C4	77.74% (139,429)	96.63% (173,309)	3.35% (6,002)	99.98% (179,311)	11.6491	54.55	50.38
ARC-Challenge	FineWeb-Edu	84.93% (93,296)	98.11% (107,769)	1.87% (2,049)	99.97% (109,818)	11.0807	31.83	28.24
	DCLM	80.32% (88,230)	97.45% (107,050)	2.53% (2,780)	99.98% (109,830)	11.3833	27.65	26.19
	C4	79.88% (87,745)	97.05% (106,607)	2.89% (3,178)	99.94% (109,785)	11.4708	24.49	20.99

Appendix

Table 13:

1.33\mathrm{B}

and

402\mathrm{M}

models trained on the same

26\mathrm{B}

-token subsets. Conventions as in Table 12 . The coverage and entropy columns apply to both models.

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	83.88% (150,436)	98.03% (175,827)	1.94% (3,484)	99.98% (179,311)	11.1923	57.00
	DCLM	78.33% (140,481)	97.18% (174,302)	2.80% (5,029)	99.99% (179,331)	11.5566	51.56
	C4	77.74% (139,425)	96.63% (173,317)	3.33% (5,978)	99.97% (179,295)	11.6480	45.16
	OpenWebText	76.05% (136,394)	96.09% (172,350)	3.88% (6,962)	99.98% (179,312)	11.7504	42.97
ARC-Challenge	FineWeb-Edu	84.93% (93,296)	98.11% (107,769)	1.87% (2,049)	99.97% (109,818)	11.0808	23.81
	DCLM	80.12% (88,016)	97.40% (106,994)	2.58% (2,835)	99.98% (109,829)	11.3923	21.59

Appendix

Table 14:

402\mathrm{M}

model,

8.5\mathrm{B}

-token subset 1. Conventions as in Table 12 . Source for the

8.5\mathrm{B}

subsets in Figure 3 .

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	83.93% (150,539)	98.10% (175,941)	1.88% (3,380)	99.98% (179,321)	11.1955	56.00
	DCLM	78.63% (141,023)	97.27% (174,461)	2.72% (4,870)	99.99% (179,331)	11.5345	51.01
	C4	77.74% (139,427)	96.63% (173,312)	3.34% (5,995)	99.97% (179,307)	11.6487	46.21
ARC-Challenge	FineWeb-Edu	85.01% (93,382)	98.11% (107,775)	1.84% (2,017)	99.95% (109,792)	11.0829	23.38
	DCLM	80.41% (88,332)	97.49% (107,090)	2.48% (2,729)	99.97% (109,819)	11.3728	21.16
	C4	79.88% (87,745)	97.05% (106,604)	2.89% (3,170)	99.93% (109,774)	11.4701	20.73

Appendix

Table 15:

402\mathrm{M}

model,

8.5\mathrm{B}

-token subset 2. Conventions as in Table 12 .

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	84.15% (150,929)	98.21% (176,149)	1.77% (3,166)	99.98% (179,315)	11.1988	56.44
	DCLM	78.55% (140,881)	97.24% (174,412)	2.74% (4,908)	99.98% (179,320)	11.5464	50.13
	C4	77.74% (139,430)	96.63% (173,307)	3.34% (5,988)	99.97% (179,295)	11.6495	46.76
ARC-Challenge	FineWeb-Edu	85.24% (93,635)	98.22% (107,895)	1.72% (1,894)	99.95% (109,789)	11.0862	24.40
	DCLM	80.32% (88,233)	97.45% (107,052)	2.51% (2,760)	99.97% (109,812)	11.3837	20.99
	C4	79.89% (87,760)	97.05% (106,612)	2.87% (3,151)	99.92% (109,763)	11.4708	19.88

Appendix

Table 16:

402\mathrm{M}

model,

8.5\mathrm{B}

-token subset 3. Conventions as in Table 12 .

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	84.07% (150,785)	98.16% (176,062)	1.81% (3,253)	99.98% (179,315)	11.1948	56.31
	DCLM	78.62% (141,009)	97.27% (174,461)	2.71% (4,863)	99.98% (179,324)	11.5415	50.76
	C4	77.74% (139,431)	96.63% (173,318)	3.34% (5,986)	99.97% (179,304)	11.6484	46.89
ARC-Challenge	FineWeb-Edu	85.15% (93,533)	98.16% (107,827)	1.79% (1,965)	99.95% (109,792)	11.0826	23.63
	DCLM	80.41% (88,332)	97.47% (107,074)	2.50% (2,746)	99.97% (109,820)	11.3788	22.18
	C4	79.88% (87,742)	97.05% (106,610)	2.87% (3,157)	99.93% (109,767)	11.4695	20.14

Appendix

Table 17:

402\mathrm{M}

model,

8.5\mathrm{B}

-token subset 4. Conventions as in Table 12 .

Task	Dataset	$\sim$ 80%	$\sim$ 95%	95–100%	Seen	Entropy	Score
ARC-Easy	FineWeb-Edu	83.85% (150,383)	98.11% (175,963)	1.87% (3,353)	99.98% (179,316)	11.1883	56.00
	DCLM	78.50% (140,792)	97.19% (174,316)	2.79% (5,009)	99.98% (179,325)	11.5521	50.34
	C4	77.74% (139,430)	96.63% (173,307)	3.34% (5,988)	99.97% (179,295)	11.6494	47.18
ARC-Challenge	FineWeb-Edu	84.91% (93,273)	98.13% (107,792)	1.83% (2,006)	99.95% (109,798)	11.0771	23.38
	DCLM	80.27% (88,171)	97.41% (107,004)	2.54% (2,795)	99.95% (109,799)	11.3884	21.16
	C4	79.89% (87,754)	97.05% (106,603)	2.88% (3,162)	99.92% (109,765)	11.4704	20.73

Appendix

Table 18:

402\mathrm{M}

model,

8.5\mathrm{B}

-token subset 5. Conventions as in Table 12 .

Task	Accuracy order	UCE	Bigram CE	Ppl	Cosine	TF–IDF
ARC-E	FwEdu $>$ DCLM $>$ C4	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
ARC-C	FwEdu $>$ DCLM $>$ C4	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
HellaSwag	C4 $>$ DCLM $>$ FwEdu	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$
MMLU	FwEdu $>$ DCLM $>$ C4	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$
SciQ	FwEdu $>$ DCLM $>$ C4	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
OBQA	FwEdu $>$ DCLM $>$ C4	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$

Appendix

Table 19: Corpus-ranking agreement per task (a

\checkmark

means the measure reproduces the trained-accuracy ordering of the three corpora).

	FineWeb-Edu	DCLM	C4	OpenWebText
FineWeb-Edu	0.000196	0.0506	0.0447	0.0557
DCLM	0.0506	0.000282	0.0325	0.0228
C4	0.0447	0.0325	0.000204	0.0380
OpenWebText	0.0557	0.0228	0.0380	0.001490

Appendix

Table 20: Pairwise Jensen–Shannon divergence between corpus unigram distributions (bits). Diagonals (italic) are split-half noise floors.

Benchmark	Compared across (rising UCE)	UCE (bits)	Score
BLiMP (grammar, $1.33$ B)	DCLM / C4 / FineWeb-Edu	$12.42$ / $12.66$ / $12.87$	$61.0$ / $80.5$ / $80.9$
MathQA (arithmetic, $1.33$ B)	DCLM / FineWeb-Edu / C4	$11.34$ / $11.34$ / $11.41$	$23.3$ / $23.3$ / $22.1$
Global-PIQA (non-English, $3.36$ B)	Fr / Es / De / Zh / Ar	$19.8$ / $20.9$ / $22.5$ / $27.2$ / $28.8$	$46$ / $50$ / $58$ / $49$ / $52$

Appendix

Table 21: Capability families outside the scope of UCE. Entries within each row are ordered by rising UCE.

Source	Model	Metric	Random	DSIR-paper	UCE-Filter	BETR
DCLM	$3.33$ B	Targeted	$62.95$ $\pm 0.03$	$63.01$ $\pm 0.33$	$\mathbf{66.26}$ $\pm 0.20$	$65.53$ $\pm 0.34$
		Out-of-target	$48.75$ $\pm 0.53$	$46.23$ $\pm 0.38$	$\mathbf{49.56}$ $\pm 0.09$	$43.65$ $\pm 0.25$
		Overall	$57.78$ $\pm 0.17$	$56.91$ $\pm 0.35$	$\mathbf{60.19}$ $\pm 0.14$	$57.57$ $\pm 0.15$
	$1.34$ B	Targeted	$58.85$ $\pm 0.56$	$59.85$ $\pm 0.53$	$62.49$ $\pm 0.26$	$\mathbf{62.96}$ $\pm 0.43$
		Out-of-target	$\mathbf{46.57}$ $\pm 0.53$	$43.79$ $\pm 0.23$	$46.22$ $\pm 0.94$	$41.35$ $\pm 0.46$
		Overall	$54.38$ $\pm 0.54$	$54.01$ $\pm 0.28$	$\mathbf{56.58}$ $\pm 0.23$	$55.10$ $\pm 0.25$

Appendix

Table 22: Zero-shot accuracy (%) of the UCE-filter and of embedding-based selection (BETR) with Random and DSIR-paper, across sources and scales. Mean

\pm

standard deviation over

3

training seeds (

1234

42

7777

), under the per-task metric convention of Table 32 . Targeted averages the

7

tasks used for selection (ARC-Easy, ARC-Challenge, HellaSwag, SciQ, OpenBookQA, PIQA, SWAG), Out-of-target the

4

other tasks (LAMBADA, CommonsenseQA, BoolQ, SocialIQA), and Overall all

11

. Bold marks the best mean in each row. The UCE-filter column is shaded.

Target	Source	Random	DSIR [ 43 ]	BETR [ 45 ]	UCE-Filter
CSR7	DCLM	$11.17$	$11.08$	$11.08$	$\mathbf{10.92}$
CSR7	FineWeb-Edu	$11.32$	$11.20$	$11.25$	$\mathbf{10.93}$
OOT-4	DCLM	$11.39$	$11.43$	$11.79$	$11.41$
OOT-4	FineWeb-Edu	$11.62$	$11.65$	$11.93$	$11.58$

Appendix

Table 23: Corpus-to-benchmark UCE (bits

\downarrow

) on the selected subsets against the

7

-task targeted suite (CSR7) and the

4

out-of-target tasks (OOT-4), with the BETR selection added to the methods of Table 1 . OOT-4 values are measured with the pipeline of Appendix A.1 on a fixed-seed sample of each subset, so absolutes are not comparable across the CSR7 and OOT-4 blocks. Bold marks the lowest CSR7 value in each row. The UCE-filter column is shaded.

Selection rule	vs CSR7	vs OOT-4
order- $3$ , $\beta=10$	$-0.183$	$-0.004$
order- $3$ , $\beta=10^{2}$	$-0.194$	$+0.006$
order- $3$ , $\beta=10^{3}$	$-0.202$	$+0.009$
order- $3$ , $\beta=10^{4}$	$-0.209$	$+0.013$
UCE-filter (unigram)	$\mathbf{-0.274}$	$+0.016$

Appendix

Table 24: Selected-subset UCE relative to the random control (bits), for the conditional

3

-gram filter across smoothing strengths

\beta

and for the unigram UCE-filter.

Selection rule	Targeted	Out-of-target	Overall
order- $3$ , $\beta=10$	$62.39$	$46.26$	$56.53$
UCE-filter (unigram)	$62.39$	$47.11$	$56.84$
Random	$58.26$	$45.97$	$53.79$
DSIR-paper	$59.42$	$44.06$	$53.84$

Appendix

Table 25: Zero-shot accuracy (%) of the trained order-

3

(

\beta=10

) selection and of the paper’s rules (

1.34

B, DCLM, one matched training seed). The three-seed standard deviations of these arms in Table 2 reach

0.56

pp on the targeted average and

0.94

pp on the out-of-target average. Targeted averages the seven selection targets, Out-of-target the four held-out tasks, Overall all eleven.

Source	Selection target	CSR7 ( $7$ )	$T_{B}$ ( $3$ )
DCLM	CSR7	$66.26$	$59.44$
	$T_{B}$	61.27 $\pm$ 0.13	59.62 $\pm$ 0.44
	Random	$62.95$	$58.42$
FineWeb-Edu	CSR7	$64.64$	$53.91$
	$T_{B}$	61.13 $\pm$ 0.20	54.46 $\pm$ 0.25
	Random	$62.80$	$52.61$

Appendix

Table 26: Held-out target transfer (

3.33

B). Each row is a from-scratch model trained on a UCE-filter selection targeting either the CSR7 suite or the held-out target

T_{B}

(BoolQ, SocialIQA, LAMBADA), or a row-count-matched random control, evaluated on both targets under the per-task metric convention of Table 32 . CSR7 averages its

7

tasks and

T_{B}

its

3

. All cells are means over three training seeds (

1234

42

7777

), evaluated in one pass, and the

T_{B}

rows additionally report

\sigma

Target $Q$	Reference $P$	Coverage (%)	$D_{\mathrm{KL}}(Q\\|P)$	UCE $H(Q,P)$
CSR7	CSR7 (self)	$100.00$	$0.000$	$10.21$
CSR7	OOT-4	$97.77$	$1.008$	$11.21$
OOT-4	OOT-4 (self)	$100.00$	$0.000$	$10.73$
OOT-4	CSR7	$96.02$	$1.509$	$12.24$

Appendix

Table 27: Benchmark-group cross-UCE (bits

\downarrow

). Word-level unigram cross-entropy between the two benchmark groups, with no training corpus involved. The target

Q

sets the vocabulary and the marginal, and the reference

P

supplies counts over that vocabulary under the zero-count smoothing of Appendix A.1 . Coverage is the share of the target’s probability mass whose words occur in the reference. Diagonal rows reduce to the marginal entropy

H(Q)

Group	Task	UCE vs CSR7	UCE vs OOT-4
CSR7	ARC-Challenge	$10.87$	$11.16$
	ARC-Easy	$11.01$	$11.36$
	HellaSwag	$10.24$	$11.13$
	OpenBookQA	$11.18$	$11.36$
	PIQA	$10.45$	$11.39$
	SciQ	$11.46$	$11.82$

Appendix

Table 28: Per-task cross-UCE (bits

\downarrow

). Each task’s word distribution measured against the two group marginals, which include the task itself, so closeness to the own group is partly built in for the mass-dominant tasks (see text).

Retention	DCLM	FineWeb-Edu
$5\%$	$10.769$	$10.814$
$10\%$	$10.779$	$10.841$
$20\%$ (paper)	$10.801$	$10.883$
$40\%$	$10.836$	$10.944$
$60\%$	$10.871$	$10.997$
$80\%$	$10.929$	$11.069$

Appendix

Table 29: Selected-subset UCE against CSR7 (bits

\downarrow

) as a function of the retention rate, measured on fixed-seed samples of each subset, the convention of the OOT-4 block of Table 1 . Absolute values sit slightly below that table’s full-scan CSR7 column, so within-column deltas are the quantity of interest. The

100\%

row is the unfiltered corpus.

Scale	Source $\times$ Method	Mean (Avg)	$\sigma_{\text{select}}$ (pp)	$\sigma_{\text{train}}$ (pp)
$1.34\mathrm{B}$	DCLM $\times$ UCE-filter	$62.30$	$0$ (det.)	$0.26$
	DCLM $\times$ DSIR-paper	$60.20$	$0.56$	$0.53$
	DCLM $\times$ Random	$59.36^{\dagger}$	n/a ( $n{=}1$ )	$0.56$
	FineWeb-Edu $\times$ UCE-filter	$61.68$	$0$ (det.)	$0.15$
	FineWeb-Edu $\times$ DSIR-paper	$59.84$	$0.48$	$0.08$
	FineWeb-Edu $\times$ Random	$58.95$	$0.26$	$0.41$

Appendix

Table 30: Targeted-suite mean and

\sigma_{\text{select}}

from selection seeds

\{42,100,200\}

at training seed

42

, alongside

\sigma_{\text{train}}

reproduced from Table 2 (training seeds

\{1234,42,7777\}

at the original selection seed). The UCE-filter is selection-deterministic and the column reports its value at training seed

42

CSR7 accuracy (%)	UCE-filter	Random	DSIR-paper
Original	$62.39$	$58.26$	$59.42$
Decontaminated and retrained	$62.17$	$58.77$	$59.88$

Appendix

Table 31: Decontaminate, re-select, and retrain from scratch (

1.34

B, DCLM, same-seed comparisons).

Targeted (UCE-filter targets)

Out-of-target

Average

Source

Method

ARC-E

ARC-C

HSwag

SciQ

OBQA

PIQA

SWAG

LAMB

CSQA

BoolQ

SIQA

Targeted

Overall

1.34B parameters

DCLM

UCE-filter (

p{=}0.20

)

68.90

\pm

0.26

38.17

\pm

0.43

66.84

\pm

0.09

90.47

\pm

0.60

40.27

\pm

1.17

77.86

\pm

0.55

54.96

\pm

0.17

64.34

\pm

0.21

19.38

\pm

1.17

57.84

\pm

2.53

43.31

\pm

0.64

62.49

\pm

0.26

56.58

\pm

0.23

Random (

p{=}0.20

)

62.93

\pm

1.39

34.22

\pm

0.31

60.58

\pm

0.45

89.60

\pm

0.85

37.53

\pm

0.42

74.90

\pm

0.72

52.19

\pm

0.14

60.67

\pm

0.12

20.78

\pm

0.24

62.02

\pm

0.97

42.80

\pm

1.10

58.85

\pm

0.56

54.38

\pm

0.54

DSIR-paper [ 43 ]

65.17

\pm

1.88

35.84

\pm

1.49

63.00

\pm

0.34

88.10

\pm

0.78

38.80

\pm

0.92

77.02

\pm

0.54

50.99

\pm

0.06

51.73

\pm

0.24

19.49

\pm

2.28

61.89

\pm

1.97

42.05

\pm

0.78

59.85

\pm

0.53

54.01

\pm

0.28

BETR [ 45 ]

71.25

\pm

1.13

43.34

\pm

1.19

67.10

\pm

0.08

91.67

\pm

1.50

40.87

\pm

0.83

78.75

\pm

0.41

47.75

\pm

0.16

43.02

\pm

0.73

19.44

\pm

0.55

61.28

\pm

1.49

41.64

\pm

0.50

62.96

\pm

0.43

55.10

\pm

0.25

Appendix

Table 32: Per-task zero-shot accuracy (%) for the UCE-filter runs of Section 5.2 and for BETR [ 45 ] with the same CSR7 targets, reported as mean

\pm\sigma

across three independent training seeds (

1234

42

7777

). The left block holds the seven targeted tasks that define the target distribution

Q

, and the middle block the four out-of-target tasks. Bold marks the best mean among {UCE-filter, Random, DSIR-paper, BETR} within each cell. The DSIR-LN row (italics) is the single-seed length-normalized ablation of Appendix E.2 and is excluded from the bold comparison.

Scale	Source	Method	Targeted	Out-of-target	Overall
$1.34$ B	DCLM	UCE-filter	$\mathbf{64.50}$	$\mathbf{46.84}$	$\mathbf{58.08}$
		Random	$61.22$	$45.80$	$55.61$
		DSIR-paper	$61.69$	$41.83$	$54.47$
$1.34$ B	FineWeb-Edu	UCE-filter	$\mathbf{63.74}$	$\mathbf{42.23}$	$\mathbf{55.92}$
		Random	$61.20$	$40.54$	$53.69$
		DSIR-paper	$61.65$	$39.20$	$53.49$

Appendix

Table 33: Suite-level

5

-shot accuracy (%), three-seed means. Targeted averages the seven selection targets, Out-of-target the four held-out tasks, Overall all eleven.

Targeted (UCE-filter targets)

Out-of-target

Average

Source

Method

ARC-E

ARC-C

HSwag

SciQ

OBQA

PIQA

SWAG

LAMB

CSQA

BoolQ

SIQA

Targeted

Overall

1.34B parameters

DCLM

UCE-filter (

p{=}0.20

)

74.40

\pm

0.28

42.32

\pm

0.81

67.88

\pm

0.36

94.60

\pm

0.60

41.00

\pm

1.22

77.95

\pm

0.54

53.35

\pm

0.15

59.04

\pm

1.39

19.82

\pm

0.75

61.86

\pm

0.87

46.66

\pm

0.51

64.50

\pm

0.26

58.08

\pm

0.30

Random (

p{=}0.20

)

69.95

\pm

0.37

38.40

\pm

0.67

61.20

\pm

0.18

93.93

\pm

0.98

38.73

\pm

2.31

75.21

\pm

0.60

51.11

\pm

0.05

55.62

\pm

0.68

20.17

\pm

0.90

61.55

\pm

4.31

45.87

\pm

0.37

61.22

\pm

0.66

55.61

\pm

0.17

DSIR-paper [ 43 ]

70.74

\pm

0.84

39.36

\pm

0.91

63.32

\pm

0.34

92.37

\pm

0.21

39.13

\pm

2.10

77.02

\pm

0.33

49.91

\pm

0.13

44.76

\pm

1.57

19.85

\pm

1.07

58.71

\pm

3.94

44.01

\pm

0.82

61.69

\pm

0.17

54.47

\pm

0.47

DSIR-LN (ablation)

73.11

43.52

68.22

94.60

40.80

78.62

52.29

50.63

21.38

58.47

44.37

64.45

56.91

Appendix

Table 34: Per-task

5

-shot accuracy (%) for the UCE-filter runs of Section 5.2 , reported as mean

\pm\sigma

across three independent training seeds (

1234

42

7777

). Conventions follow Table 32 .

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

May 29, 2026

Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

Various language domains have undergone remarkable changes in recent years; these shifts are largely attributed to the advent of Large Language Models and their misalignment with natural language usage. These misalignments are thought to partly originate in the preference-learning stage, e.g. Reinforcement Learning from Human Feedback, which generally makes models more useful but simultaneously may introduce systematic lexical bias. In terms of lexical behavior, this is visible in a model's preference for certain formats or the overuse of words (delve, furthermore), even when such patterns are not present in base model outputs. Research on lexical misalignment induced during preference training is constrained by reliance on manual curation. We address this, by introducing the Triangulated Preference Shift score, a metric that triangulates between human gold standards, base models, and instruct variants to isolate shifts induced specifically by preference learning, without manual curation. We provide data across six model families, anchor the results in the literature, and illustrate the general approach's utility by analyzing whether preference learning shifts models toward what could be interpreted as a "language of prestige". The metric provides an initial automated method to quantify behavioral shifts attributable to preference tuning, and thus, may help inform model alignment and development of trustworthy AI.

Xiaoyang Ming, Jose Hernandez, Thomas Stephan Juzek

Florida State University

LLM Alignment LLM Fine-Tuning Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Jun 2, 2026

Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models

The language used by digital chat assistants such as ChatGPT can diverge from human expectations (misalignment). Research, mostly on Scientific English, has described both what divergences occur and, to some extent, why, linking them to the training stage of human preference learning. Yet, existing approaches rely on manual curation. This paper introduces two curation-free, assumption-light evaluation metrics: the Lexical Alignment Score, which identifies lexical overuse, and the Triangulated Preference Shift, which quantifies how much of such shifts can be attributed to human preference learning. Using PubMed abstracts, continuations were generated and measured using windowed document prevalence across six model families (Falcon, Gemma, Llama, Mistral, OLMo, Yi). The procedure identifies, without manual intervention, overused items such as 'suggest', 'additionally', and 'strategy', and estimates their link to preference learning. Our findings replicate prior work and remain stable across parameter settings, random seeds, and evaluation on further data. The approach scales readily and enables systematic study of lexical (mis)alignment beyond Scientific English and across languages, and as such, the metrics have the potential to contribute to improved alignment for future models and understanding of its origins.

Thomas Stephan Juzek, Xiaoyang Ming, Jose A. Hernandez

Florida State University

LLM Evaluation LLM Alignment Preference Alignment Text Generation

Ask PDF

Mar 19, 2026

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing/evaluating alignment methods.

Yana Veitsman, Yihong Liu, Hinrich Schütze

University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)

Cross-Lingual Representation Alignment Cross-Lingual Transfer Multilingual Language Models Gradient Interference

Ask PDF

Abstract

Language Model Pretraining

Explore similar work

Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders