Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

October 10, 2024

Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, Sanjiv Kumar
Context Learning Gradient Descent Multi Layer Gradient Flow Layer Transformer
Plug-and-Play Performance Estimation for LLM Services without Relying on Labeled Data
Can Wang, Dianbo Sui, Hongliang Sun, Hao Ding, Bolin Zhang, Zhiying Tu
Large Language Model Context Learning Plug and Play Supervised Baseline Labeled Data
DemoShapley: Valuation of Demonstrations for In-Context Learning
Shan Xie, Man Luo, Chadly Daniel Stern, Mengnan Du, Lu Cheng
Context Learning Shot Learning Noisy Demonstration Context Demonstration Private Valuation Demonstration Example

October 9, 2024

Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning
Zhengyu Hu, Yichuan Li, Zhengyu Chen, Jingang Wang, Han Liu, Kyumin Lee, Kaize Ding
Large Language Model Graph Neural Network Context Learning Graph Structured Data Text Attributed Graph Context Graph
Retrieval-Augmented Decision Transformer: External Memory for In-context RL
Thomas Schmied, Fabian Paischer, Vihang Patil, Markus Hofmarcher, Razvan Pascanu, Sepp Hochreiter
Reinforcement Learning Context Learning Memory Capacity Retrieval Enhanced Transformer
Tree of Problems: Improving structured problem solving with compositionality
Armel Zebaze, Benoît Sagot, Rachel Bawden
Context Learning Chain of Thought Complex Reasoning Task Related Problem Structured Document Compositional Language Step by Step
MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data
Mingu Kang, Dongseok Lee, Woojin Cho, Jaehyeon Park, Kookjin Lee, Anthony Gruber, Youngjoon Hong, Noseong Park
Large Language Model Neural Network Context Learning Foundational Model Noise Prior Advection Diffusion Reaction

October 8, 2024

Vector-ICL: In-context Learning with Continuous Vector Representations
Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, Jianfeng Gao
Large Language Model Context Learning Different Context Vector Representation Continuous Representation Language Modeling Objective Context Vector
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition
Zheyang Xiong, Ziyang Cai, John Cooper, Albert Ge, Vasilis Papageorgiou, Zack Sifakis, Angeliki Giannou, Ziqian Lin, Liu Yang, Saurabh Agarwal, Grigorios G Chrysos, Samet Oymak, Kangwook Lee, Dimitris Papailiopoulos
Large Language Model Context Learning Different Context First Order Superposition Task Vector Multiple Task

October 6, 2024

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI
Chengyuan Xu, Radha Kumaran, Noah Stier, Kangyou Yu, Tobias Höllerer
Context Learning 3D Object Detection Multimodal Phenomenon Multimodal Representation 3D Environment Augmented Reality Application Spatial Intelligence
GAMformer: In-Context Learning for Generalized Additive Models
Andreas Mueller, Julien Siems, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter
Context Learning Generalized Additive Model Shape Function Gam Depth
Revisiting In-context Learning Inference Circuit in Large Language Models
Hakaze Cho, Mariko Kato, Yoshihiro Sakai, Naoya Inoue
Language Model Context Learning

October 3, 2024

Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference
Wei Cheng, Tianlu Wang, Yanmin Ji, Fan Yang, Keren Tan, Yiyu Zheng
Large Language Model Context Learning Calibration Performance Label Free Discriminative Reply Calibrated Prediction
GraphIC: A Graph-Based In-Context Example Retrieval Model for Multi-Step Reasoning
Jiale Fu, Yaqing Wang, Simeng Han, Jiaming Fan, Chen Si, Xu Yang
Context Learning Multi Step Reasoning Context Example AI Reasoning

Context Learning

Papers

Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?

Plug-and-Play Performance Estimation for LLM Services without Relying on Labeled Data

DemoShapley: Valuation of Demonstrations for In-Context Learning

Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning

Retrieval-Augmented Decision Transformer: External Memory for In-context RL

Tree of Problems: Improving structured problem solving with compositionality

MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

Vector-ICL: In-context Learning with Continuous Vector Representations

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

Transformers learn variable-order Markov chains in-context

Task Diversity Shortens the ICL Plateau

Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

GAMformer: In-Context Learning for Generalized Additive Models

Revisiting In-context Learning Inference Circuit in Large Language Models

Enhanced Transformer architecture for in-context learning of dynamical systems

In-context Learning in Presence of Spurious Correlations

RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning

Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference

GraphIC: A Graph-Based In-Context Example Retrieval Model for Multi-Step Reasoning