Context Learning

In-context learning (ICL) is a paradigm shift in machine learning, focusing on enabling models to adapt to new tasks using only a few examples provided within the input, without requiring parameter updates. Current research emphasizes understanding ICL's mechanisms, particularly within transformer-based large language models, and improving its effectiveness through techniques like enhanced example selection, chain-of-thought prompting, and addressing issues such as spurious correlations and copy bias. This research is significant because ICL offers a more efficient and adaptable approach to many machine learning problems, impacting fields ranging from natural language processing and computer vision to scientific computing and beyond.

Papers

January 11, 2024

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning
Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, Jinming Wen
Large Language Model Language Model Context Learning Backdoor Attack Attack Success Rate Inherent Vulnerability

January 10, 2024

Leveraging Print Debugging to Improve Code Generation in Large Language Models
Xueyu Hu, Kun Kuang, Jiankai Sun, Hongxia Yang, Fei Wu
Large Language Model Context Learning Code Generation Code Generation Task Code Debugging

January 7, 2024

Grimoire is All You Need for Enhancing Large Language Models
Ding Chen, Shichao Song, Qingchen Yu, Zhiyu Li, Wenjin Wang, Feiyu Xiong, Bo Tang
Large Language Model Language Model Context Learning Chinese CodEx

January 5, 2024

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks
Kevin Everson, Yile Gu, Huck Yang, Prashanth Gurunath Shivakumar, Guan-Ting Lin, Jari Kolehmainen, Ivan Bulyko, Ankur Gandhe, Shalini Ghosh, Wael Hamza, Hung-yi Lee, Ariya Rastrow, Andreas Stolcke
Large Language Model Context Learning Speech Recognition Spoken Language Understanding Natural Language Understanding

December 28, 2023

Improving In-context Learning via Bidirectional Alignment
Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding, Shafiq Joty
Large Language Model Context Learning Bidirectional Alignment

December 27, 2023

How Robust are LLMs to In-Context Majority Label Bias?
Karan Gupta, Sumegh Roychowdhury, Siva Rajesh Kasa, Santhosh Kumar Kasa, Anish Bhanushali, Nikhil Pattisapu, Prasanna Srinivasa Murthy
Large Language Model Context Learning Contextual Biasing Label Bias Context Sample

December 26, 2023

December 21, 2023

On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning
Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, Ivan Vulić
Large Language Model Self Supervised Context Learning Supervised Fine Tuning Context Example Model Calibration Task Performance

December 20, 2023

Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interest
Emily Groves, Minhong Wang, Yusuf Abdulle, Holger Kunz, Jason Hoelscher-Obermaier, Ronin Wu, Honghan Wu
Fine Tuning Context Learning Supervised Learning Automatic Curation BiOmedical Ontology Biomedical Knowledge

December 19, 2023

Can Transformers Learn Sequential Function Classes In Context?
Ryan Campbell, Emma Guo, Evan Hu, Reya Vir, Ethan Hsiao
Transformer Megatron Decepticons Context Learning Transformer Model Context Information Function Class

December 17, 2023

kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning
Wenting Zhao, Ye Liu, Yao Wan, Yibo Wang, Qingyang Wu, Zhongfen Deng, Jiangshu Du, Shuaiqi Liu, Yunlong Xu, Philip S. Yu
Context Learning Semantic Parsing Natural Language Prompt KNN Clip

December 16, 2023

Stochastic Bayesian Optimization with Unknown Continuous Context Distribution via Kernel Density Estimation
Xiaobin Huang, Lei Song, Ke Xue, Chao Qian
Context Learning Optimization Purpose Bayesian Optimization Kernel Density Context Distribution

December 15, 2023

Low-resource classification of mobility functioning information in clinical sentences using large language models
Tuan Dung Le, Thanh Duong, Thanh Thieu
Large Language Model Context Learning MAESTRO Dataset Full Information Task Specific Clinical Text Mobility Management Low Resource Text Classification

December 14, 2023

December 13, 2023

Reconciling Shared versus Context-Specific Information in a Neural Network Model of Latent Causes
Qihong Lu, Tan T. Nguyen, Qiong Zhang, Uri Hasson, Thomas L. Griffiths, Jeffrey M. Zacks, Samuel J. Gershman, Kenneth A. Norman
Context Learning Context Information Neural Network Model LLM Inference Latent Causal Sharing Matter Contextual Structure Common Cause

December 12, 2023

Context Learning

Papers

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning

Leveraging Print Debugging to Improve Code Generation in Large Language Models

Grimoire is All You Need for Enhancing Large Language Models

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Improving In-context Learning via Bidirectional Alignment

How Robust are LLMs to In-Context Majority Label Bias?

Dynamic In-Context Learning from Nearest Neighbors for Bundle Generation

Supervised Knowledge Makes Large Language Models Better In-context Learners

On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning

Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interest

Can Transformers Learn Sequential Function Classes In Context?

kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning

Stochastic Bayesian Optimization with Unknown Continuous Context Distribution via Kernel Density Estimation

Low-resource classification of mobility functioning information in clinical sentences using large language models

A Comparative Analysis of Fine-Tuned LLMs and Few-Shot Learning of LLMs for Financial Sentiment Analysis

MotherNet: A Foundational Hypernetwork for Tabular Classification

Reconciling Shared versus Context-Specific Information in a Neural Network Model of Latent Causes

Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization

Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection

ICL Markup: Structuring In-Context Learning using Soft-Token Tags