cs.AIOct 24, 2026
SaveData Optimization for LLMs: A Survey
Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, China
Abstract
Large language models (LLMs) have attracted great attention in both academic and industrial communities in recent years. Numerous survey studies have covered various aspects of LLMs, such as learning strategies, applications, alignment, explainability, and evaluation. However, little headway has been made in reviewing data-related aspects for LLMs, despite data being recognized as a key factor contributing to the emergence of the LLMs era. A wide range of data optimization algorithms and techniques are employed in LLMs' training and application processes. These techniques often originate from diverse deep learning domains, with their theoretical inspirations and heuristic motivations appearing unrelated to each other. This study aims to develop a comprehensive framework for LLMs' data optimization techniques to enhance data utilization and efficiency. We first discuss data challenges in LLMs' training and inference. Second, we summarize data perception dimensions for evaluating LLMs' data properties. These dimensions primarily include quality, difficulty, diversity, and redundancy, laying a foundation for subsequent optimization. Third, existing optimization techniques are reviewed and categorized into four groups: data selection, augmentation, reweighting, and others. Fourth, we analyze interconnections between these categories. Finally, we identify key challenges and future directions for LLMs' data optimization compared with conventional deep learning approaches.
Figures & tables
Figure 1. Nine issues around the data in conventional deep learning (left) ( Wu and Yao, 2024 ) , and around the data in LLMs (right). Key mappings: (1) “Distribution drift” (conventional DL) corresponds to “Distribution mismatch” (LLMs)—with LLM-specific extensions like inference-time prompt distribution shift; (2) “Noisy” (conventional DL) aligns with “Low quality” (LLMs)—including LLM-unique formatting chaos; (3) “Biased distribution” (conventional DL) maps to “Imbalance” (LLMs)—including novel multi-domain mixture proportion bias. Remaining issues (e.g., data poisoning) are LLM-specific or extended from conventional DL.
| Survey studies | Topics | Techniques | Stages | Perception |
| Albalak et al. ( Albalak et al., 2025 ) , Wang et al. ( Wang et al., 2024c ) | Data selection | Selection, re-ordering | Training, Inference | Mentioned but not systematically elaborated |
| Ding et al. ( Ding et al., 2024a ) , Chai et al. ( Chai et al., 2026 ) | Data augmentation | Augmentation, synthesis | Training, Inference | Not referred |
| Wang et al. ( Wang et al., 2024d ) | Synthesis, augmentation | Synthesis, augmentation | Training, Inference | Mentioned but not systematically elaborated |
| Das et al. ( Das et al., 2025 ) , Zhao et al. ( Zhao et al., 2025 ) | Security, poisoning | Selection | Training, Inference | Not referred |
| Navigli et al. ( Navigli et al., 2023 ) , Chu et al. ( Chu et al., 2024 ) | Bias and fairness | Selection, augmentation, reweighting | Training, Inference | Data balance |
| Ferdaus et al. ( Ferdaus et al., 2024 ) , Huang et al. ( Huang et al., 2023 ) | Trustworthiness | Selection, augmentation, reweighting | Training, Inference | Data quality, balance |
Table 1. Several relevant data-centric survey studies (most are about LLMs).
Figure 2. The main goals (left) and pipeline of data optimization for LLMs (Data synthesis, when performed without corpus anchoring, is excluded from this pipeline; when used to augment existing data, it is subsumed under data augmentation.)
Figure 3. Impact intensity of data optimization techniques on key LLM aspects. Reading guide: Rows represent six technical categories; columns represent four LLM aspects. Dark blue (High) indicates the primary target of the technique; medium blue (Medium) indicates significant secondary effects; light blue (Low) indicates minor or indirect effects. For example, Data Selection is primarily designed for Effectiveness and Efficiency, yet attains High trustworthiness because quality filtering removes noisy samples that cause hallucinations.
| Relationship type | Dimension pair | Description |
| Synergy | Quality Diversity | Jointly promote trustworthiness and fairness |
| Synergy | Difficulty Value | Hard samples often carry high marginal value |
| Correlation | Uncertainty Quality | Label noise increases predictive entropy |
| Correlation | Uncertainty Difficulty | Harder samples show higher model uncertainty |
| Correlation | Redundancy Value | Near-duplicates exhibit low Shapley value |
| Tension | Quality vs. Diversity | Aggressive filtering erodes tail-domain coverage |
Table 2. Inter-dimensional dependencies, correlations, and tensions.
| Dimension | Operational Definition | Common Quantitative Proxies | Connection to LLM Performance |
| Quality | The extent to which data is free from noise, errors, and formatting inconsistencies | Perplexity, domain-specific vocabulary ratio, readability scores, annotation agreement | If neglected: model replicates noise, generates factual errors, and degrades instruction-following; if optimized: enhances factual accuracy and output coherence ( Gao et al., 2024a ; Havrilla and Iyer, 2024 ) |
| Difficulty | The level of challenge a sample presents to the model’s learning or inference process | Loss-based (training/validation loss), perplexity, reasoning steps length, reference model accuracy | If too low: model resorts to simple pattern matching, failing on complex tasks; if too high: training becomes unstable and convergence difficult; if optimized: elicits complex reasoning and prevents premature convergence ( Chen et al., 2024 ; Gao et al., 2025 ; Muennighoff et al., 2025 ) |
| Diversity | The degree of variation and coverage across samples in a dataset | Vendi score (entropy of similarity matrix eigenvalues), cluster counts, pairwise distances, lexical diversity | If insufficient: fails on minor domains/styles/groups, overfits to training patterns; if optimized: improves cross-domain generalization and reduces spurious correlations ( Havrilla et al., 2024 ; Li et al., 2025f ; Lu et al., 2024 ) |
| Uncertainty | The level of ambiguity in model predictions or noise in training labels | Predictive entropy, semantic consistency across samples, LLM self-reported confidence | If unquantified: overconfident in wrong answers, unaware of knowledge boundaries; if leveraged: enables abstention or human review, mitigating hallucinations ( Farquhar et al., 2024 ; Kadavath et al., 2022 ; Shorinwa et al., 2025 ) |
| Value | The marginal contribution of a sample to model performance | Shapley value approximations, influence functions, gradient-based influence, training-free proxies | If neglected: training inefficient, compute wasted on uninformative samples; if optimized: achieves comparable performance with less data, enables precise demonstration selection ( Kwon et al., 2024 ; Sachdeva et al., 2024 ; Xia et al., 2024 ) |
| Redundancy | The presence of highly similar or duplicated content across samples | N-gram overlap, embedding similarity (e.g., within epsilon balls), information gain scores | If unaddressed: model memorizes rather than learns generalizable patterns; if optimized: improves both efficiency and generalization ( Cunha et al., 2025 ; Tirumala et al., 2023 ; Yin et al., 2024 ) |
Table 3. Data perception dimensions: definitions, proxies, and relevance to LLM performance.
Figure 4. A taxonomy for data perception for LLMs. Reading guide: The tree is read from root (left) to leaves (right). Each branch under “Data Perception” corresponds to one of the eight dimensions discussed in Section 4. Second-level nodes indicate measurement families (e.g., “Loss-based” under Difficulty), and leaf nodes cite representative methods. The taxonomy is operational: it maps abstract dimensions to concrete, computable proxies.
Figure 5. An illustrative end-to-end pipeline of data optimization for LLMs.
| Technique Category | Pretraining | SFT | RLHF / Alignment | Inference | Typical Pipeline |
| Data Selection | Heuristic filtering (language, length), quality scoring, domain reweighting, deduplication ( Xie et al., 2023 ; Tirumala et al., 2023 ; Ankner et al., 2024 ) | Difficulty-aware selection, diversity-aware sampling, influence-based selection ( Gao et al., 2025 ; Xia et al., 2024 ; Li et al., 2024b ) | Preference data selection, contrastive sampling, quality filtering for preference pairs ( Zhou et al., 2025 ) | Demonstration selection for ICL, example ordering optimization ( Guo et al., 2024 ; Wu et al., 2024 ) | Pretraining: heuristic filter quality score domain reweight; SFT: difficulty-aware + diversity sampling |
| Data Augmentation | Synthetic data generation, back-translation, paraphrasing ( Abdin et al., 2024 ) | Instruction evolution, response rewriting, counterfactual generation ( Ding et al., 2024a ; Chen et al., 2023c ) | Preference pair augmentation, rejection sampling, self-play generation ( Zhu et al., 2024a ) | RAG, query rewriting ( Lewis et al., 2020 ) | Pretraining: synthetic data for low-resource domains; SFT: instruction evolution + back-translation |
| Data Reweighting | Domain-wise reweighting, temperature sampling, importance weighting ( Fan et al., 2024b ; Chung et al., 2023 ) | Sample-wise reweighting based on difficulty or value, loss adjustment ( Jiang et al., 2024 ; Chen et al., 2024 ) | Preference weighting, reward modeling, importance sampling for alignment ( Zhou et al., 2024b ) | Attention reweighting, context compression weighting ( Jiang et al., 2023 ) | Pretraining: domain temperature; SFT: sample difficulty weighting; RLHF: preference pair weighting |
| Other Techniques | Dataset distillation (emerging), curriculum ordering ( Nguyen et al., 2025a ; Lee et al., 2024 ) | Prompt compression for efficient fine-tuning, data merging ( Cai et al., 2025a ) | Preference distillation, alignment tuning ( Cai and others, 2025 ) | Prompt compression, test-time data optimization ( Jiang et al., 2023 ; Li et al., 2023b ; Chen et al., 2025a ) | Pretraining: curriculum ordering; Inference: prompt compression + RAG |
Table 4. Data optimization techniques across the LLM lifecycle: applicability, inherent impacts, and typical pipelines.
| Lifecycle stage | Typical evaluation goals | Representative tasks | Common metrics |
| Pre-training | Improve corpus quality, data efficiency, transferability, and convergence behavior | Language modeling validation; downstream transfer evaluations; domain adaptation; multilingual and code pretraining settings | Perplexity; validation loss; downstream accuracy/F1; convergence speed; training stability; data efficiency |
| Supervised fine-tuning (SFT) | Improve task performance, instruction-following ability, robustness, and generalization | Question answering, reasoning, summarization, coding, dialogue, and domain-specific instruction datasets | Accuracy; F1; ROUGE/BLEU; pass@k; win rate; robustness/generalization scores |
| Alignment | Improve helpfulness, harmlessness, preference alignment, and safety | red-teaming test; safety evaluation; harmlessness and refusal test | Preference win rate; reward score; violation/refusal rate; safety score; human evaluation |
| Inference-time optimization | Improve response quality under latency, context, or token-budget constraints | Prompt compression, retrieval/reranking, long-context evaluation, and test-time adaptation settings | Latency; token cost; memory usage; response quality; efficiency–utility trade-off |
Table 5. Representative evaluation typical goals, tasks, and metrics for assessing data optimization methods across the LLM lifecycle.
| LLMs | Selection | Augmentation | Reweighting | Re-ordering | Perception |
| Llama 3 ( Grattafiori et al., 2024 ) | ✓ | ✓ | X | ✓ | Data quality, difficulty, diversity, redundancy, contamination |
| Qwen 2.5 ( Qwen et al., 2025 ) | ✓ | ✓ | X | ✓ | Data quality, diversity, redundancy, contamination |
| Salamandra ( Gonzalez-Agirre et al., 2025 ) | ✓ | X | X | X | Data quality, diversity, redundancy |
| InternLM2 ( Cai et al., 2024 ) | ✓ | X | ✓ | X | Data quality, difficulty, redundancy, contamination |
| GLM ( GLM, 2024 ) | ✓ | X | ✓ | X | Data quality, diversity, redundancy |
| Phi-4 ( Abdin et al., 2024 ) | ✓ | ✓ | ✓ | ✓ | Data quality, difficulty, diversity, redundancy, contamination |
Table 6. Data optimization explicitly mentioned in the technical reports of several recent foundation models.
| Technical Category | # Papers | Representative Anchors |
| Data Selection | 42 | [2, 4, 58, 97, 121, 139, 173, 207, 238] |
| Data Augmentation | 31 | [19, 21, 29, 43, 47, 115, 182, 191, 253] |
| Data Reweighting | 19 | [27, 34, 50, 73, 78, 87, 124, 275] |
| Re-ordering | 9 | [45, 69, 112, 174, 232] |
| Prompt Compression | 11 | [14, 33, 88, 131, 138, 159, 170] |
| Dataset Distillation | 7 | [16, 114, 151, 163, 202, 212] |
Table 7. Quantitative census of surveyed papers across major technical categories (Section 5). Only papers receiving substantive methodological discussion (at least one paragraph) are counted; peripheral citations are excluded.
Figure 6. A taxonomy for data optimization techniques for LLMs. Reading guide: The root “Data Optimization for LLMs” branches into four principal categories (Selection, Augmentation, Reweighting, Others), each of which further decomposes into methodological families and then into representative papers. Readers can navigate from high-level strategy to specific algorithms by following the tree from left to right.
| Optimization Objective | Typical Techniques | Rationale, Typical Scenarios, and Trade-offs |
| Accuracy First | Quality-focused data selection ( Chen et al., 2023a ; Zhang et al., 2024d ) ; Domain-specific augmentation ( Ding et al., 2024a ; Chen et al., 2023c ) ; Difficulty-aware reweighting ( Gao et al., 2025 ; Chen et al., 2024 ) | Prioritizes data fidelity and task alignment over computational efficiency. Suitable for high-stakes domains like medical diagnosis, legal analysis, and scientific reasoning where error tolerance is low. Trade-off : May increase computational cost; if token-constrained : prioritize quality selection over augmentation to stay within budget. |
| Efficiency First | Lightweight selection (heuristic, perplexity-based) ( Xia et al., 2025 ; Ankner et al., 2024 ) ; Dataset distillation ( Nguyen et al., 2025a ; Tao et al., 2024 ) ; Coreset selection ( Sorscher et al., 2022 ; Xia et al., 2023 ) | Aims to reduce training data volume while preserving model performance. Ideal for rapid prototyping, resource-constrained environments, or when training multiple model variants. Trade-off : May sacrifice accuracy; if accuracy non-negotiable : use difficulty-aware selection instead of random pruning. |
| Cost Saving | Data reweighting ( Jiang et al., 2024 ; Chen et al., 2024 ) ; Active learning ( Parkar et al., 2024 ) ; Influence-based selection ( Xia et al., 2024 ; Kwon et al., 2024 ) | Maximizes performance gains from limited annotation or acquisition budgets. Suitable for scenarios with expensive human labeling (e.g., medical annotation) or restricted access to third-party data. Trade-off : May underexplore rare but important cases if value estimation is imperfect; if coverage is critical : combine with diversity-aware selection or targeted augmentation. |
| Knowledge Coverage | Diversity-aware sampling ( Li et al., 2025f ; Lu et al., 2024 ) ; Retrieval-based augmentation ( Lewis et al., 2020 ) ; Domain-aware reweighting ( Fan et al., 2024b ; Ye et al., 2025a ) | Ensures broad representation across languages, domains, or tasks. Critical for multilingual models, domain adaptation, and general-purpose assistants. Trade-off : Broader coverage may dilute specialization or introduce noisier samples; if downstream precision is critical : pair with quality filtering or domain-aware selection. |
| Token Saving | Prompt compression ( Jiang et al., 2023 ; Li et al., 2023b ) ; Demonstration selection ( Guo et al., 2024 ; Xu et al., 2024c ) ; Context pruning ( Jiang et al., 2023 ) | Reduces input length without substantially sacrificing output quality. Essential for inference optimization, long-context tasks, and cost-sensitive API usage (pay-per-token). Trade-off : Aggressive compression may remove task-relevant context; if accuracy is sensitive to context : prefer selective demonstration pruning over heavy prompt compression. |
| Balanced / Multi-Objective | Hybrid pipelines (e.g., RAG + LoRA) ( Lewis et al., 2020 ; Hu et al., 2024 ) ; Pareto-aware optimization ( Xie et al., 2025b ) | Combines techniques to navigate trade-offs between competing objectives. Increasingly adopted in production systems where no single objective dominates. Trade-off : Hybrid pipelines are more flexible but also harder to tune and evaluate; practical use : they require multi-metric validation rather than a single performance score. |
Table 8. Objective-driven data optimization: typical technique matches and trade-offs.
| Technique | Pre | IT | Align | Infer | Data-Ltd | Compute-Ltd |
| Selection | High (filter, dedup, mix) ( Xie et al., 2023 ; Tirumala et al., 2023 ; Abdin et al., 2024 ) | High (diff, div, influence) ( Gao et al., 2025 ; Xia et al., 2024 ; Li et al., 2024b ) | Med (pref filter) ( Zhou et al., 2025 ) | High (demo select) ( Guo et al., 2024 ; Wu et al., 2024 ) | High (high-value) ( Xia et al., 2024 ; Kwon et al., 2024 ) | High (coreset) ( Sorscher et al., 2022 ; Xia et al., 2023 ) |
| Augmentation | Low (synthetic) ( Abdin et al., 2024 ) | High (evolve, para) ( Ding et al., 2024a ; Chen et al., 2023c ) | Med (reject sampling) ( Zhu et al., 2024a ) | High (RAG) ( Lewis et al., 2020 ) | High (generate) ( Parkar et al., 2024 ) | Low (costly) |
| Reweighting | High (domain mix) ( Fan et al., 2024b ; Chung et al., 2023 ) | Med (diff weight) ( Jiang et al., 2024 ; Chen et al., 2024 ) | High (pref weight) ( Zhou et al., 2024b ) | Low | High (max impact) ( Jiang et al., 2024 ) | Low |
| Other | Med (distill) ( Nguyen et al., 2025a ) | Med (prompt comp) ( Cai et al., 2025a ) | Low | High (prompt comp) ( Jiang et al., 2023 ; Li et al., 2023b ) | Med (distill) ( Nguyen et al., 2025a ) | High (distill, comp) ( Nguyen et al., 2025a ; Jiang et al., 2023 ) |
Table 9. Scenario-based mapping: when to apply different data optimization techniques.
Figure 7. Similarity/opposition between the main data optimization techniques.
More in this topic
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 353 papers