Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

October 18, 2023

Few-Shot In-Context Imitation Learning via Implicit Graph Alignment
Vitalis Vosylius, Edward Johns
Context Learning Imitation Learning Shot Learning Feature Imitation Graph Alignment Task Relationship
MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations
Arkil Patel, Satwik Bhattamishra, Siva Reddy, Dzmitry Bahdanau
Context Learning Long Context Natural Language Description Context Learning Ability Ambiguous Interpretation

October 17, 2023

Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning
Rui Wen, Tianhao Wang, Michael Backes, Yang Zhang, Ahmed Salem
Context Learning Privacy Policy Security Related Private Data Privacy Analysis LLM Adaptation Soft Prompt Tuning Model Stealing
Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue
Shiwei Zhang, Mingfang Wu, Xiuzhen Zhang
Large Language Model Case Study Context Learning Annotated Dataset Metadata Information Scientific Data LLM in Context

October 16, 2023

IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, Tongliang Liu
Context Learning Context Example Selective Annotation
In-context Pretraining: Language Modeling Beyond Document Boundaries
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, Mike Lewis
Large Language Model Language Model Context Learning Pre Trained Long Context Document Boundary
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, Yu Bai
Large Language Model LeArning Abstract Transformer Megatron Decepticons Context Learning Transformer Architecture Meaningful Representation Simple Function Different Context Representation Function
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning
Anirudh Som, Karan Sikka, Helen Gent, Ajay Divakaran, Andreas Kathol, Dimitra Vergyri
Context Learning Noisy Demonstration Paraphrase Generation Offensive Content Hard Paraphrase
Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs
Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, Chris Parnin
Large Language Model Self Supervised Context Learning Impact Assessment Table Representation Approximate Operator
Generative Calibration for In-context Learning
Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, Kang Liu
Large Language Model Context Learning Context Aware Context Modeling Generative Calibration
A Search for Prompts: Generating Structured Answers from Contracts
Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, François Longtin
Context Learning Search Query Complex Prompt Semantic Matching Human Agreement Generative Question Answering Prompt Template Structured Answer

October 15, 2023

AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents
Jake Grigsby, Linxi Fan, Yuke Zhu
Reinforcement Learning Context Learning Adaptive Agent
In-Context Learning with Iterative Demonstration Selection
Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, Wenming Ye
Context Learning Shot Example Demonstration Selection Zero Shot Chain of Thought
Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming
Manh Hung Nguyen, Sebastian Tschiatschek, Adish Singla
Context Learning BEHAVIOR Explanation Open Ended Student Friendly Knowledge Student Model Visual Programming Learning Resource Student Response
Large Language Model-Aware In-Context Learning for Code Generation
Jia Li, Ge Li, Chongyang Tao, Jia Li, Huangzhao Zhang, Fang Liu, Zhi Jin
Large Language Model Context Learning Code Generation LLM in Context

October 12, 2023

Do pretrained Transformers Learn In-Context by Gradient Descent?
Lingfeng Shen, Aayush Mishra, Daniel Khashabi
Language Model Transformer Megatron Decepticons Pre Trained Language Model Context Learning Gradient Descent Different Context Training Model
How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, Peter L. Bartlett
Context Learning Multi Task Linear Regression Ridge Regression Unseen Task Statistical Complexity Linear Attention Model

Context Learning

Papers

Few-Shot In-Context Imitation Learning via Implicit Graph Alignment

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning

Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue

IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models

In-context Pretraining: Language Modeling Beyond Document Boundaries

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning

Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs

Generative Calibration for In-context Learning

A Search for Prompts: Generating Structured Answers from Contracts

AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents

In-Context Learning with Iterative Demonstration Selection

Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming

Large Language Model-Aware In-Context Learning for Code Generation

SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation

Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning

In-Context Learning for Few-Shot Molecular Property Prediction

Do pretrained Transformers Learn In-Context by Gradient Descent?

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?