Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

October 24, 2023

October 22, 2023

O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models
Yuchen Xiao, Yanchao Sun, Mengda Xu, Udari Madhushani, Jared Vann, Deepeka Garg, Sumitra Ganesh
Large Language Model Context Learning Mutual Distillation Sequential Decision Data Driven Discovery Decision Making Benchmark

October 21, 2023

Ensemble-Instruct: Generating Instruction-Tuning Data with a Heterogeneous Mixture of LMs
Young-Suk Lee, Md Arafat Sultan, Yousef El-Kurdi, Tahira Naseem Asim Munawar, Radu Florian, Salim Roukos, Ramón Fernandez Astudillo
Language Model Context Learning Raw Data Tuned Lm Heterogeneous Medium Self Instruct

October 20, 2023

October 19, 2023

October 18, 2023

October 17, 2023

October 16, 2023

Context Learning

Papers

In-Context Learning Creates Task Vectors

POE: Process of Elimination for Multiple Choice Reasoning

O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models

Ensemble-Instruct: Generating Instruction-Tuning Data with a Heterogeneous Mixture of LMs

Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning

Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens

Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks

Measuring Pointwise $\mathcal{V}$-Usable Information In-Context-ly

Few-Shot In-Context Imitation Learning via Implicit Graph Alignment

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning

Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue

IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models

In-context Pretraining: Language Modeling Beyond Document Boundaries

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning

Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs

Generative Calibration for In-context Learning