Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

May 26, 2023

A Closer Look at In-Context Learning under Distribution Shifts
Kartik Ahuja, David Lopez-Paz
Context Learning Distribution Shift Glance Annotation

May 24, 2023

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing
Shufan Wang, Sebastien Jean, Sailik Sengupta, James Gung, Nikolaos Pappas, Yi Zhang
Context Learning Semantic Parsing Task Oriented Task Oriented Dialogue Measurement System Task Constraint Constraint Compliance Conversational Semantic
Boosting Cross-lingual Transferability in Multilingual Models via In-Context Learning
Sunkyoung Kim, Dayeon Ki, Yireun Kim, Jinsik Lee
Context Learning Multilingual Model Cross Lingual Transfer Cross Lingual Transferability
Adversarial Demonstration Attacks on Large Language Models
Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, Chaowei Xiao
Large Language Model Context Learning Adversarial Perspective Unseen Validation Adversarial Demonstration
Coverage-based Example Selection for In-Context Learning
Shivanshu Gupta, Matt Gardner, Sameer Singh
Large Language Model Context Learning Context Example Information Redundancy BERTScore Metric
Estimating Large Language Model Capabilities without Labeled Test Data
Harvey Yiyun Fu, Qinyuan Ye, Albert Xu, Xiang Ren, Robin Jia
Language Model Context Learning Test Data Accuracy Estimation
EXnet: Efficient In-context Learning for Data-less Text classification
Debaditya Shome, Kuldeep Yadav
Language Model Context Learning Zero Shot Learning Large Pre Trained Language Model Cross Task Generalization Efficient in Context Learning

May 23, 2023

RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning
Alexander Scarlatos, Andrew Lan
Reinforcement Learning Context Learning Large Pre Trained Language Model Context Example Supervised Baseline
Active Learning Principles for In-Context Learning with Large Language Models
Katerina Margatina, Timo Schick, Nikolaos Aletras, Jane Dwivedi-Yu
Context Learning Active Learning Shot Learning Context Example Pool Based Active Learning
Skill-Based Few-Shot Selection for In-Context Learning
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, Jian-Guang Lou
Large Language Model Context Learning Shot Learning View Selection Skill Mention KNN Seq
In-Context Probing: Toward Building Robust Classifiers via Probing Large Language Models
Afra Amini, Massimiliano Ciaramita
Large Language Model Context Learning Classification Task Robust Classifier
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, Xu Sun
Large Language Model Context Learning Information Flow Textual Label Label Embeddings Visual Information Anchor
Dr.ICL: Demonstration-Retrieved In-context Learning
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, Vincent Y Zhao
Large Language Model Context Learning Task Demonstration Retrieval Augmented in Context Learning
Learning Relevant Contextual Variables Within Bayesian Optimization
Julien Martinelli, Ayush Bharti, Armi Tiihonen, S. T. John, Louis Filstroff, Sabina J. Sloman, Patrick Rinke, Samuel Kaski
Context Learning Bayesian Optimization Probabilistic Surrogate Context Optimization Contextual Bayesian Optimization Contextual Variable
CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation
Aswanth Kumar, Ratish Puduppully, Raj Dabre, Anoop Kunchukuttan
Large Language Model Language Model Context Learning Machine Translation Translation Quality Context Example Multiple Feature
Make a Choice! Knowledge Base Question Answering with In-Context Learning
Chuanyuan Tan, Yuehe Chen, Wenbiao Shao, Wenliang Chen
Context Learning Knowledge Base Multiple Choice Value Laden Choice KBQA Model KBQA Datasets
Concept-aware Training Improves In-context Learning Ability of Language Models
Michal Štefánik, Marek Kadlčík
Language Model Context Learning Context Information Context Learning Ability Context Learner

Context Learning

Papers

A Closer Look at In-Context Learning under Distribution Shifts

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing

Boosting Cross-lingual Transferability in Multilingual Models via In-Context Learning

Adversarial Demonstration Attacks on Large Language Models

Coverage-based Example Selection for In-Context Learning

Estimating Large Language Model Capabilities without Labeled Test Data

EXnet: Efficient In-context Learning for Data-less Text classification

RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning

Active Learning Principles for In-Context Learning with Large Language Models

Skill-Based Few-Shot Selection for In-Context Learning

In-Context Probing: Toward Building Robust Classifiers via Probing Large Language Models

Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

Dr.ICL: Demonstration-Retrieved In-context Learning

Learning Relevant Contextual Variables Within Bayesian Optimization

CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation

Make a Choice! Knowledge Base Question Answering with In-Context Learning

Concept-aware Training Improves In-context Learning Ability of Language Models

Small Language Models Improve Giants by Rewriting Their Outputs

Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations

Friendly Neighbors: Contextualized Sequence-to-Sequence Link Prediction