Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

November 18, 2024

Re-examining learning linear functions in context
Omar Naim, Guilhem Fouilhé, Nicholas Asher
Training Data Full Model Context Learning Transformer Based Model Context Information Linear Function

November 17, 2024

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, Yuke Zhu
Reinforcement Learning Transformer Megatron Decepticons Context Learning Multi Task Meta Reinforcement Learning Multi Task Optimization

November 16, 2024

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason M. Klusowski, Jianqing Fan, Mengdi Wang
Context Learning Transformer Model Context Information Layer Transformer Softmax Attention

November 15, 2024

On the Privacy Risk of In-context Learning
Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, Franziska Boenisch
Large Language Model Context Learning Shot Learning Membership Inference Attack Privacy Risk Natural Language Prompt

November 14, 2024

StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
Dilxat Muhtar, Yelong Shen, Yaming Yang, Xiaodong Liu, Yadong Lu, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Xueliang Zhang, Jianfeng Gao, Weizhu Chen, Qi Zhang
Large Language Model Context Learning Test Time Adaptation Efficient Inference Inference Cost

November 13, 2024

XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL
Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, Yu Li
Large Language Model Context Learning Entity Recognition Text to SQL

November 12, 2024

Decision Feedback In-Context Symbol Detection over Block-Fading Channels
Li Fan, Jing Yang, Cong Shen
Context Learning Pre Trained Transformer Human Decision Symbol Detection Large Scale Fading Transformer Based Decoder

November 10, 2024

In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messages
Joseph Gatto, Parker Seegmiller, Timothy E. Burdick, Sarah Masud Preum
New Framework Context Learning Patient Friendly Language Sensitive Patient Healthcare Workflow Patient Privacy Synthetic Patient Physician Dialogue

November 8, 2024

WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning
Xiangyu Zhao, Zhiwang Zhou, Wenlong Zhang, Yihao Liu, Xiangyu Chen, Junchao Gong, Hao Chen, Ben Fei, Shiqi Chen, Wanli Ouyang, Xiao-Ming Wu, Lei Bai
Foundation Model Context Learning Image Translation Weather Forecasting Climate Data Visual Foundation Model

November 7, 2024

November 6, 2024

November 5, 2024

Mixtures of In-Context Learners
Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, Pasquale Minervini
Context Learning Mixture Component Noisy Demonstration Different Context Transformer Based LLM Context Learner

November 4, 2024

Context Learning

Papers

Re-examining learning linear functions in context

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context

On the Privacy Risk of In-context Learning

StreamAdapter: Efficient Test Time Adaptation from Contextual Streams

XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

Decision Feedback In-Context Symbol Detection over Block-Fading Channels

In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messages

WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning

Adversarial Robustness of In-Context Learning in Transformers for Linear Regression

Vision Language Models are In-Context Value Learners

Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences

Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks

EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

Mixtures of In-Context Learners

Fair In-Context Learning via Latent Concept Variables

TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos

Pretrained transformer efficiently learns low-dimensional target functions in-context

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

Shortcut Learning in In-Context Learning: A Survey