Context Learning
In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.
Papers
On the Noise Robustness of In-Context Learning for Text Generation
Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, Hongxin Wei
Transformer In-Context Learning for Categorical Data
Aaron T. Wang, Ricardo Henao, Lawrence Carin
Benchmarking General-Purpose In-Context Learning
Fan Wang, Chuan Lin, Yang Cao, Yu Kang
Unifying Demonstration Selection and Compression for In-Context Learning
Jun Gao, Ziqiang Cao, Wenjie Li
Automatic Domain Adaptation by Transformers in In-Context Learning
Ryuichiro Hataya, Kota Matsui, Masaaki Imaizumi
ARC: A Generalist Graph Anomaly Detector with In-Context Learning
Yixin Liu, Shiyuan Li, Yu Zheng, Qingfeng Chen, Chengqi Zhang, Shirui Pan
Mixture of In-Context Prompters for Tabular PFNs
Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, Wei Wang
Unsupervised Meta-Learning via In-Context Learning
Anna Vettoruzzo, Lorenzo Braccaioli, Joaquin Vanschoren, Marlena Nowaczyk
Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars
Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, Bryan Kian Hsiang Low
Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning
Simon Yu, Jie He, Pasquale Minervini, Jeff Z. Pan
MLPs Learn In-Context on Regression and Classification Tasks
William L. Tong, Cengiz Pehlevan
Synergizing In-context Learning with Hints for End-to-end Task-oriented Dialog Systems
Vishal Vivek Saley, Rocktim Jyoti Das, Dinesh Raghu, Mausam
Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMs
Siyuan Guo, Aniket Didolkar, Nan Rosemary Ke, Anirudh Goyal, Ferenc Huszár, Bernhard Schölkopf
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, Reynold Cheng
Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
Shang Liu, Zhongze Cai, Guanting Chen, Xiaocheng Li
Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning
Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, Shangtong Zhang
DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning
Zijian Zhou, Xiaoqiang Lin, Xinyi Xu, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low
Why In-Context Learning Transformers are Tabular Data Classifiers
Felix den Breejen, Sangmin Bae, Stephen Cha, Se-Young Yun