Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

Ankit Bhattacharjee; Indrasabarna Hari; Krityapriya Bhaumik; Arka Das; Piyush Sarangi

cs.CLOct 12, 2026

Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

Authors: Ankit Bhattacharjee, Indrasabarna Hari, Krityapriya Bhaumik, Arka Das, Piyush Sarangi

Organizations: Indian Institute of Technology Kharagpur, India · Asutosh College, University of Calcutta, India

Abstract

While language models provide powerful mechanisms for evaluating textual predictability, it remains unclear whether the relative extents of LM surprisal remain stable across models. This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources. We consider two stylistic environments: a corpus of contemporary poetry and a corpus of academic research papers. We demonstrate that mean and variance of cross-entropy maintain robust cross-model rank correlation ( $ρ> 0.90$ ) across both regimes. To test whether this agreement merely reflects generic differences in textual complexity, we compare it against six parameter-free compression baselines. The extent of agreement among language models is found to be substantially higher than that among model-baselines, for most of the measures. Geometric analysis of the structurally constrained academic regime reveals that higher-order moments are sensitive to specific models, although strong agreement persists. The findings suggest that relative extents of textual unpredictability may be independent of the language model they are computed against.

Figures & tables

Panel of Figure 1 — Figure 1: Pairwise Spearman’s rank correlation ( $\rho$ ) of cross-entropy moments across the four evaluated language models for the poetic regime. The near-perfect agreement for mean and variance demonstrates the invariance of the metric, demonstrating that the metric survives across different model paradigms.

Panel of Figure 2 — Figure 2: Pairwise Spearman’s rank correlation ( $\rho$ ) of cross-entropy moments across the four evaluated language models for the academic regime. Despite the extreme stylistic variance compression enforced by institutional formatting, first two moments (mean and variance) maintain robust invariance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model pair	Mean BPC	Variance	Skewness	Excess kurtosis
GPT-2 – Pythia	[0.919, 0.995]	[0.814, 0.996]	[0.778, 0.985]	[0.775, 0.959]
GPT-2 – Qwen	[0.914, 0.993]	[0.787, 0.993]	[0.464, 0.947]	[0.588, 0.926]
GPT-2 – TinyLlama	[0.898, 0.992]	[0.923, 0.994]	[0.832, 0.992]	[0.646, 0.948]
Pythia – Qwen	[0.974, 0.999]	[0.945, 0.998]	[0.663, 0.970]	[0.762, 0.975]
Pythia – TinyLlama	[0.922, 0.999]	[0.684, 0.996]	[0.691, 0.968]	[0.641, 0.951]
Qwen – TinyLlama	[0.897, 1.000]	[0.655, 0.997]	[0.318, 0.913]	[0.458, 0.935]

Appendix

Table 1: 95% paired-bootstrap confidence intervals for Spearman’s rank correlation coefficients across the four measures in the poetic corpus (

n=28

poets).

Model pair	Mean BPC	Variance	Skewness	Excess kurtosis
GPT-2 – Pythia	[0.803, 0.989]	[0.812, 0.982]	[0.460, 0.914]	[0.407, 0.855]
GPT-2 – Qwen	[0.772, 0.986]	[0.776, 0.969]	[0.234, 0.916]	[0.462, 0.857]
GPT-2 – TinyLlama	[0.846, 0.984]	[0.765, 0.963]	[0.349, 0.906]	[0.528, 0.888]
Pythia – Qwen	[0.882, 0.991]	[0.784, 0.974]	[0.696, 0.947]	[0.664, 0.945]
Pythia – TinyLlama	[0.880, 0.989]	[0.824, 0.973]	[0.791, 0.977]	[0.790, 0.962]
Qwen – TinyLlama	[0.927, 0.994]	[0.934, 0.993]	[0.803, 0.978]	[0.820, 0.976]

Appendix

Table 2: 95% paired-bootstrap confidence intervals for Spearman’s rank correlation coefficients across the four measures in the academic corpus (

n=29

papers).

Moment	Compressor	$\bar{\rho}_{\mathrm{comp\text{-}LM}}$	$\Delta$	95% CI of $\Delta$
Mean ( $\bar{\rho}_{\mathrm{LM\text{-}LM}}=0.963$ , $n=28$ )
	plain deflate	0.505	0.458	[0.156, 0.806]
	plain LZMA	0.520	0.443	[0.166, 0.781]
	plain bzip2	0.580	0.383	[0.184, 0.661]
	plain zstd	0.579	0.384	[0.170, 0.677]
	primed deflate	0.768	0.195	[0.056, 0.431]

Appendix

Table 3: Compressor baselines versus language models, poetic corpus. † : interval contains zero.

Moment	Compressor	$\bar{\rho}_{\mathrm{comp\text{-}LM}}$	$\Delta$	95% CI of $\Delta$
Mean ( $\bar{\rho}_{\mathrm{LM\text{-}LM}}=0.975$ , $n=29$ )
	plain deflate	0.463	0.512	[0.194, 0.908]
	plain LZMA	0.379	0.596	[0.200, 1.019]
	plain bzip2	0.355	0.620	[0.269, 1.026]
	plain zstd	0.363	0.612	[0.260, 1.014]
	primed deflate	0.700	0.275	[0.130, 0.508]

Appendix

Table 4: Compressor baselines versus language models, academic corpus.

Model	Checkpoint identifier
GPT-2	gpt2
Pythia-410M	EleutherAI/pythia-410m
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B
TinyLlama-1.1B	TinyLlama/TinyLlama-1.1B-Chat-v1.0

Appendix

Table 5: Language-model checkpoints used in the experiments. All models were used for language model scoring purpose.

Baseline	Implementation setting
Plain DEFLATE	zlib ; raw DEFLATE stream, level 9
Plain LZMA	lzma ; raw LZMA2 stream, preset 9 with the extreme preset
Plain bzip2	bz2 ; compression level 9
Plain Zstandard	zstandard ; compressor level 19
Primed DEFLATE	Difference between compressed reference-plus-chunk size and compressed reference size, using raw DEFLATE level 9
Primed LZMA	The analogous size difference using raw LZMA2, preset 9 with the extreme preset

Appendix

Table 6: Compression implementations and settings used as non-neural baselines.

Component	Version
Python (analysis / compression)	3.13.15 / 3.10.0
PyTorch	2.11.0+cu128
PyMuPDF	1.28.2
Transformers	$\geq 4.45$ , $<5$ (constraint)
NumPy, pandas, SciPy	Not recorded
Matplotlib, Accelerate	Not recorded

Appendix

Table 7: Software versions recorded in the notebooks.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 504 papers

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Sep 23, 2026Kaifeng Tan, Yudong Li, Linlin ShenLLM Evaluation Data Contamination in Language Models
Shenzhen University · Tsinghua University
Forecasting Downstream Performance of LLMs With Proxy Metrics
May 18, 2026Arkil Patel, Siva Reddy, Marius Mosbach +1Language Model Pretraining LLM Evaluation
Mila – Quebec AI Institute & McGill University Canada · ServiceNow Research · Periodic Labs
The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions
Mar 29, 2026Jarosław HryszkoLanguage Modeling
Institute of Computer Science, Faculty of Mathematics and Computer Science, Jagiellonian University, Krak´ow, Poland

Sep 23, 2026J/K move · Enter open · S save

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

Kaifeng Tan, Yudong Li, Linlin Shen

Shenzhen University · Tsinghua University

LLM Evaluation Data Contamination in Language Models

Open paper AskPDF arXiv

Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction-following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model's predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero-shot MMLU accuracy. Our results yield three main findings: (1) compression performance follows a consistent scaling trend with model size; (2) attention-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and (3) lower compression rates are strongly associated with higher zero-shot MMLU accuracy. Code is available at https://github.com/Jellyfish042/uncheatable_eval.

Abstract

Language Modeling

Explore similar work

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

Forecasting Downstream Performance of LLMs With Proxy Metrics

The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions