Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

August 8, 2023

August 7, 2023

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning
Hunter McNichols, Wanyong Feng, Jaewook Lee, Alexander Scarlatos, Digory Smith, Simon Woodhead, Andrew Lan
Context Learning Multiple Choice Question High Quality Distractors Feedback Generation Generative Evaluation

July 31, 2023

Scaling Sentence Embeddings with Large Language Models
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang
Large Language Model Context Learning Sentence Embeddings

July 28, 2023

Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning
Xindi Wang, Yufei Wang, Can Xu, Xiubo Geng, Bowen Zhang, Chongyang Tao, Frank Rudzicz, Robert E. Mercer, Daxin Jiang
Large Language Model Context Learning Supervised Learning Consistent Comparison Behaviour Learning Downstream Task Performance Label Perturbation

July 27, 2023

July 21, 2023

OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples
Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki
Context Learning Text Generation Fewer Example LLM Generated Text

July 15, 2023

SINC: Self-Supervised In-Context Learning for Vision-Language Tasks
Yi-Syuan Chen, Yun-Zhu Song, Cheng Yu Yeo, Bei Liu, Jianlong Fu, Hong-Han Shuai
Self Supervised Context Learning Vision Language Task Large Pre Trained Transformer Context Learning Ability

July 14, 2023

Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
Large Language Model LeArning Abstract Context Learning Context Example Dense Retriever

July 5, 2023

July 4, 2023

Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking
Brendan King, Jeffrey Flanigan
Context Learning Shot Learning Diverse Set Dialogue State Tracking Task Oriented Dialogue

July 3, 2023

Trainable Transformer in Transformer
Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, Sanjeev Arora
Language Model Transformer Based Pre Trained Language Model Context Learning Large Pre Trained Language Model

July 1, 2023

InstructEval: Systematic Evaluation of Instruction Selection Methods
Anirudh Ajith, Chris Pan, Mengzhou Xia, Ameet Deshpande, Karthik Narasimhan
Context Learning Tutorial Review Instruction Induction Instruction Data Selection

June 30, 2023

GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models
Pawan Kumar Rajpoot, Ankur Parikh
Natural Language Processing Context Learning GPT Neo Unstructured Data Knowledge Discovery Financial Relation Extraction

June 29, 2023

DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios
Vinicius G. Goecks, Nicholas R. Waytowich
Context Learning Action Sequence Disaster Response Disaster Response Scenario

June 26, 2023

June 16, 2023

Schema-learning and rebinding as mechanisms of in-context learning and emergence
Sivaramakrishnan Swaminathan, Antoine Dedieu, Rajkumar Vasudeva Raju, Murray Shanahan, Miguel Lazaro-Gredilla, Dileep George
Context Learning Path Breaking Emergence Functional Mechanism Transformer Based Large Language Model Context Dependent Sequence Prediction Learning Resource Schema Induction

Context Learning

Papers

FLIRT: Feedback Loop In-context Red Teaming

Large Language Model Prompt Chaining for Long Legal Document Classification

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

Scaling Sentence Embeddings with Large Language Models

Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning

Exploiting the Potential of Seq2Seq Models as Robust Few-Shot Learners

Metric-Based In-context Learning: A Case Study in Text Simplification

OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples

SINC: Self-Supervised In-Context Learning for Vision-Language Tasks

Learning to Retrieve In-Context Examples for Large Language Models

Scaling In-Context Demonstrations with Structured Attention

In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trick

Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking

Trainable Transformer in Transformer

InstructEval: Systematic Evaluation of Instruction Selection Methods

GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models

DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios

Understanding In-Context Learning via Supportive Pretraining Data

Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

Schema-learning and rebinding as mechanisms of in-context learning and emergence