cs.CLFeb 19, 2026

Entropy-Based Data Selection for Language Models

Authors: Hongming Li, Yang Liu, Chao Huang

Organizations: School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China · National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence, Beijing 100086, China · Shunde Innovation School, University of Science and Technology Beijing, Foshan 528399, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern language models (LMs) increasingly require two critical resources: computational resources and data resources. Data selection techniques can effectively reduce the amount of training data required for fine-tuning LMs. However, their effectiveness is closely related to computational resources, which always require a high compute budget. Owing to the resource limitations in practical fine-tuning scenario, we systematically reveal the relationship between data selection and uncertainty estimation of selected data. Although large language models (LLMs) exhibit exceptional capabilities in language understanding and generation, which provide new ways to alleviate data scarcity, evaluating data usability remains a challenging task. This makes efficient data selection indispensable. To mitigate these issues, we propose Entropy-Based Unsupervised Data Selection (EUDS) framework. Empirical experiments on sentiment analysis (SA), topic classification (Topic-CLS), and question answering (Q&A) tasks validate its effectiveness. EUDS establishes a computationally efficient data-filtering mechanism. Theoretical analysis and experimental results confirm the effectiveness of our approach. EUDS significantly reduces computational costs and improves training time efficiency with less data requirement. This provides an innovative solution for the efficient fine-tuning of LMs in the compute-constrained scenarios.

Figures & tables

Fig. 1: Overview of our framework.
Fig. 2: The impact of different entropy intervals on model performance.
Entropy LevelNumerical IntervalAbbreviation
Low Entropy(0.0,3.0)(0.0,3.0) ; (3.0,5.0)(3.0,5.0) ; (0.0,5.0)(0.0,5.0)0-3; 3-5; 0-5
Medium Entropy(0.0,8.0)(0.0,8.0) ; (3.0,10.0)(3.0,10.0) ; (3.0,8.0)(3.0,8.0)0-8; 3-10; 3-8
High Entropy(5.0,8.0)(5.0,8.0) ; (8.0,10.0)(8.0,10.0) ; (5.0,10.0)(5.0,10.0)5-8; 8-10; 5-10
TABLE I: Entropy Level Classification with Interval Ranges.
Dataset (Entropy Type)Data TypeEntropy IntervalOriginal CountSelected Count
MMLU (IE)OriData3–1020001038
SynData3–82000669
SumData8–1040002013
JoSelData3–1040001720
MMLU (GE)OriData5–1020001200
SynData5–1020001170
TABLE II: Selected Subset Sizes for Representative Datasets Under Different Entropy Types and Data Configurations. Each Case Uses the Best-Performing Entropy Interval.
DatasetSource
SA
SST-2stanfordnlp/sst2
SST-5SetFit/sst5
IMDbstanfordnlp/imdb
YelpYelp/yelp_review_full
Topic-CLS
TABLE III: Overview of raw dataset statistics
Fig. 3: The Effectiveness of EUDS in SA
Fig. 4: The Effectiveness of EUDS in Q&A
Fig. 5: The Effectiveness of EUDS in Topic-CLS
OriDataInformation EntropyGenerative EntropySemantic Entropy
TasksIntervalDataAccuracyF1IntervalDataAccuracyF1IntervalDataAccuracyF1
SST-20-364.20↓4.40↑4.44↑0-540.60↓2.00↑2.11↑0-813.25↓2.40↑2.35↑
SST-53-105.65↓4.40↑4.91↑0-350.75↓2.80↑2.93↑5-842.00↓7.60↑8.48↑
IMDb3-100.20↓0.00→0.00→3-533.55↓0.00→0.00→3-100.50↓0.00→0.00→
Yelp5-828.55↓0.80↑0.84↑0-320.45↓0.80↑0.92↑3-845.80↓1.60↑0.08↑
AGNews0-80.70↓2.40↑2.36↑0-80.25↓0.40↑0.38↑5-1013.50↓0.00→0.02↓
TABLE IV: The Best Interval Effect of Each Entropy After Selection of OriData.
SynDataInformation EntropyGenerative EntropySemantic Entropy
TasksIntervalDataAccuracyF1IntervalDataAccuracyF1IntervalDataAccuracyF1
SST-25-828.80↓2.00↑2.23↑5-1070.10↓2.00↑1.76↑3-852.00↓0.00→0.25↑
SST-53-811.15↓1.20↑0.50↑5-878.80↓2.80↑0.48↓8-1058.50↓1.60↑2.18↑
IMDb5-1034.10↓1.20↑1.09↑5-1074.60↓0.40↑0.41↑3-880.50↓0.40↑0.41↑
Yelp3-563.40↓0.80↑0.81↓3-554.65↓0.40↑0.48↓0-517.70↓1.20↓1.79↓
AGNews5-1011.55↓1.20↑1.12↑3-552.40↓1.20↑1.94↑3-563.40↓2.80↑3.00↑
TABLE V: The Best Interval Effect of Each Entropy After Selecting of SynData.
TasksInformation EntropyGenerative EntropySemantic Entropy
IntervalDataAccuracyF1IntervalDataAccuracyF1IntervalDataAccuracyF1
SST-20-346.82↓0.00→0.03↓5-835.85↓0.80↓0.90↓3-546.25↓0.00→0.05↓
SST-53-85.57↓0.80↓0.02↑5-839.40↓0.40↓0.25↓5-826.25↓2.00↑1.69↑
IMDb3-535.20↓3.20↑3.21↑5-837.85↓2.80↑2.81↑5-843.25↓3.20↑3.21↑
Yelp5-824.10↓1.20↓1.07↓3-1019.17↓2.40↓2.41↓3-59.50↓1.60↓1.78↓
AGNews0-85.47↓0.40↑0.41↑0-56.62↓0.80↑0.81↑5-1018.52↓0.40↓0.43↓
TABLE VI: Filtering the Effect Demonstration of the Optimal Interval for Generating SynData for an Established Task Gain.
TasksInformation EntropyGenerative EntropySemantic Entropy
IntervalDataAccuracyF1IntervalDataAccuracyF1IntervalDataAccuracyF1
SST-20-557.92↓0.40↑0.73↑3-1025.00↓1.60↓1.54↓5-854.12↓0.80↓0.84↓
SST-53-106.60↓1.20↓1.13↓0-516.00↓1.20↓0.55↓3-837.25↓0.80↑1.94↑
IMDb8-1060.27↓2.00↑2.01↑3-825.77↓3.20↑3.21↑8-1030.12↓2.40↑2.41↑
Yelp3-815.05↓2.40↓2.75↓3-859.80↓1.20↓0.99↓0-511.22↓0.80↑0.79↑
AGNews3-811.97↓0.40↓0.32↓0-517.30↓1.20↓1.30↓5-865.45↓1.60↓1.71↓
TABLE VII: Display of best results interval after screening OriData and SynData.
TasksInformation EntropyGenerative EntropySemantic Entropy
OriDataSynDataSumDataJoSelDataOriDataSynDataSumDataJoSelDataOriDataSynDataSumDataJoSelData
SST-525.00↓ 1.28↑45.00↓ 1.92↑48.00↓ 2.40↑10.22↓ 0.08↑10.88↓ 0.32↓74.11↓ 2.96↑49.91↓ 2.56↑65.36↓ 0.48↑29.40↓ 0.08↑87.18↓ 0.08↑49.99↓ 2.24↑58.30↓ 1.52↑
AGNews52.83↓ 0.00↑76.03↓ 2.88↑38.01↓ 1.72↑87.82↓ 8.08↑20.65↓ 0.24↓41.74↓ 1.60↑49.96↓ 5.12↑60.61↓ 5.04↑65.10↓ 0.40↑0.06↓ 1.12↑0.28↓ 2.24↑18.03↓ 1.84↑
MMLU51.51↓ 3.84↑85.71↓ 0.64↑42.85↓ 1.68↑89.38↓ 1.68↑70.00↓ 0.08↑90.87↓ 0.56↑45.43↓ 0.16↑62.17↓ 0.12↑40.85↓ 1.44↑87.27↓ 0.80↑49.51↓ 1.94↑72.25↓ 0.88↑
TABLE VIII: Demonstration of the effects of applying our data selection method at a larger data scale. Table showing changes in accuracy and volume of data.
TasksIG EntropyIS EntropyGS EntropyIGS Entropy
OriDataSynDataOriDataSynDataOriDataSynDataOriDataSynData
SST-522.65↓ 0.40↑29.60↓ 2.40↑32.20↓ 4.80↑53.35↓ 0.80↑64.75↓ 1.60↑24.65↓ 0.40↑45.00↓ 2.40↑31.90↓ 0.40↑
AGNews55.00↓ 0.00↑44.35↓ 0.80↑14.10↓ 0.40↓38.35↓ 1.60↑25.35↓ 0.00↑58.70↓ 1.20↑52.30↓ 0.80↓62.30↓ 2.00↑
MMLU55.35↓ 2.00↑66.70↓ 0.00↑60.00↓ 8.00↑67.50↓ 0.80↑72.35↓ 4.80↑70.35↓ 2.40↑61.25↓ 4.80↑70.95↓ 4.50↑
 SumDataJoSelDataSumDataJoSelDataSumDataJoSelDataSumDataJoSelData
SST-550.00↓ 0.40↓13.62↓ 2.80↓38.40↓ 0.80↓44.62↓ 0.80↓44.97↓ 0.80↑34.52↓ 4.00↑50.00↓ 0.40↓44.95↓ 0.80↓
TABLE IX: Application of Optimal Interval Effect of Combined Entropy to OriData and SynData Respectively. Table showing changes in accuracy and volume of data.
EntropyDatasetIntervalSubset Acc ↑ (%)Subset Reduce (%)Fullset Acc ↑ (%)Fullset Reduce (%)
IEOriData0-37.69↑51.85↓7.49↑66.56↓
SynData0-35.17↑34.10↓3.74↑16.97↓
SumData5-820.75↑45.63↓3.19↑49.59↓
JoSeData0-316.98↑42.97↓1.60↑41.76↓
GEOriData0-521.15↑60.00↓0.33↑69.06↓
SynData5-85.17↑42.95↓0.31↑90.97↓
TABLE X: The Impact of Applying the Same Entropy Interval to Both Full Dataset and Subset on Model Performance and Data Reduction.
DatasetMethodsAccuracy (%)
IMDbBiLSTM81.87
 RoBERTa-BiLSTM92.46
 TWSSenti95.00
 SELECT63.79
 IE (Ours)93.60
 GE (Ours)93.20
TABLE XI: Comparison of EUDS with recent SOTA methods on representative datasets.
Figure 17
Figure 18
Figure 19
More in this topic

Data Selection for Language Model Fine-Tuning

See all 25 papers

Explore similar work

CardsList

  1. PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning

    Jul 20, 2026Hang Zhang, Warren J. GrossLLM Fine-TuningEfficient Language Model Training

    Department of Electrical and Computer Engineering, McGill University Montreal, QC, Canada

  2. The Long-Term Effects of Data Selection in LLM Fine-Tuning

    May 28, 2026Yuxin Yang, Aoxiong Zeng, Xiangquan YangContinual Learning for LLMsLLM Fine-Tuning

    Shanghai University · East China Normal University

  3. Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey

    Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Efficient Language Model TrainingLLM Training

    Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria