Context Learning
In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.
Papers
N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs
Ilya Zisman, Alexander Nikulin, Andrei Polubarov, Nikita Lyubaykin, Vladislav Kurenkov
RuAG: Learned-rule-augmented Generation for Large Language Models
Yudi Zhang, Pei Xiao, Lu Wang, Chaoyun Zhang, Meng Fang, Yali Du, Yevgeniy Puzyrev, Randolph Yao, Si Qin, Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
DemoCraft: Using In-Context Learning to Improve Code Generation in Large Language Models
Nirmal Joshua Kapu, Mihit Sreejith
EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learning
Peide Huang, Yuhan Hu, Nataliya Nechyporenko, Daehwa Kim, Walter Talbott, Jian Zhang
Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning
Dong Shu, Mengnan Du
Toward Understanding In-context vs. In-weight Learning
Bryan Chan, Xinyi Chen, András György, Dale Schuurmans
TabDPT: Scaling Tabular Foundation Models
Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony L. Caterini
Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks
Paul Smolensky, Roland Fernandez, Zhenghao Herbert Zhou, Mattia Opper, Jianfeng Gao
In Context Learning and Reasoning for Symbolic Regression with Large Language Models
Samiha Sharlin, Tyler R. Josephson
Interpreting Affine Recurrence Learning in GPT-style Transformers
Samarth Bhargav, Alexander Gu
Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods
Tsachi Blau, Moshe Kimhi, Yonatan Belinkov, Alexander Bronstein, Chaim Baskin