Data-Efficient Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd
EEGDM: Label-Efficient EEG Representation Learning with Generative Diffusion Model
Aug 13, 2025Jia Hong Puah, Sim Kuan Goh, Ziwei Zhang +6Electroencephalography EEG Seizure Detection
School of Artificial Intelligence and Robotics, Xiamen University Malaysia · School of Energy and Chemical Engineering, Xiamen University Malaysia · Department of Biomedical Engineering, Universiti Malaya +3
Siamese Neural Network for Label-Efficient Critical Phenomena Prediction in 3D Percolation Models
Jul 5, 2025Shanshan Wang, Dian Xu, Jianmin Shen +3Phase Transitions Siamese Neural Networks
Key Laboratory of Quark and Lepton Physics (MOE) and Institute of Particle Physics, Central China Normal University, , Wuhan, 430079, Hubei, China · School of Engineering and Technology, Baoshan University, , Baoshan, 678000, Yunnan, China · Ecole Superieure d’Informatique Electronique Automatique, , Ivry-sur-Seine, 94200, , France
Towards More Efficient, Robust, Instance-adaptive, and Generalizable Sequential Decision making
Apr 12, 2025Zhiyong WangMulti-Armed Bandits Reinforcement Learning
Department of Computer Science and Engineering, The Chinese University of Hong Kong
Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
Apr 7, 2025Taiwei Shi, Yiyang Wu, Linxin Song +2RL for Language Model Reasoning Mathematical Reasoning
University of Southern California · Carnegie Mellon University · Mohamed Bin Zayed University of Artificial Intelligence
StoCFL: A Stochastically Clustered Federated Learning Framework for Non-IID Data with Dynamic Client Participation
Mar 2, 2023Dun Zeng, Xiangjing Hu, Shiyu Liu +3Non-IID Federated Learning Clustered Federated Learning
University of Electronic Science and Technology of China · Peng Cheng Lab · Harbin Institute of Technology, Shenzhen +1
Scaling Online Complex Event Detection with Synthetic Supervision and Mamba-Based Neural Algorithmic Reasoning
Date pendingLiying Han, Gaofeng Dong, Xiaomin Ouyang +4Neural Algorithmic Reasoning Algorithmic Reasoning
Department of Electrical and Computer Engineering University of California, Los Angeles · Department of Computer Science and Engineering Hong Kong University of Science and Technology · U.S. Army DEVCOM Army Research Laboratory +1
LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Date pendingHaoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM Quantization Quantization-Aware Training
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

Aug 13, 2025, cs.LGJ/K move · Enter open · S save

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai+10

1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd

CoT Distillation Language Model Distillation Algorithmic Reasoning LLM Reasoning Data-Efficient Learning

Open paper AskPDF arXiv

Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving. Recent methods have improved reasoning through expanded corpus and multistage training combining reinforcement learning and supervised fine-tuning. Although some methods suggest that small but targeted dataset can incentivize reasoning via only distillation, a reasoning scaling laws is still taking shape, increasing computational costs. To address this, we propose a data-efficient distillation framework (DED) that optimizes the Pareto frontier of reasoning distillation. Inspired by the on-policy learning and diverse roll-out strategies of reinforcement learning, the key idea of our approach is threefold: (1) We identify that benchmark scores alone do not determine an effective teacher model. Through comprehensive comparisons of leading reasoning LLMs, we develop a method to select an optimal teacher model. (2) While scaling distillation can enhance reasoning, it often degrades out-of-domain performance. A carefully curated, smaller corpus achieves a balanced trade-off between in-domain and out-of-domain capabilities. (3) Diverse reasoning trajectories encourage the student model to develop robust reasoning skills. We validate our method through evaluations on mathematical reasoning (AIME 2024/2025, MATH-500) and code generation (LiveCodeBench), achieving state-of-the-art results with only 0.8k carefully curated examples, bypassing the need for extensive scaling. Our systematic analysis demonstrates that DED outperforms existing methods by considering factors beyond superficial hardness, token length, or teacher model capability. This work offers a practical and efficient pathway to advanced reasoning while preserving general capabilities.

Momentum

Related topics

Latest papers 158