Context Learning Ability

In-context learning (ICL) refers to the ability of large language models (LLMs) and other deep learning architectures to perform tasks based on a few examples provided in the input, without explicit training or fine-tuning on that specific task. Current research focuses on understanding the mechanisms underlying ICL, including the role of attention mechanisms and model architecture, and on improving ICL performance through techniques like prompt engineering, data distillation, and the integration of external knowledge. This research is significant because it promises to improve the efficiency and adaptability of AI systems across diverse domains, from natural language processing and computer vision to biomedical applications and wireless communications, by reducing the reliance on extensive labeled datasets.

Papers

August 28, 2024

Leveraging Large Language Models for Wireless Symbol Detection via In-Context Learning
Momin Abbas, Koushik Kar, Tianyi Chen
Large Language Model Deep Neural Network Context Learning Symbol Detection Context Learning Ability

May 24, 2024

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
Shang Liu, Zhongze Cai, Guanting Chen, Xiaocheng Li
Strong Generalization Transformer Based Context Learning Regression Task Deep Understanding Context Learning Ability Bi Objective

February 16, 2024

Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm
Yuanzhen Xie, Xinzhou Jin, Tao Xie, MingXiong Lin, Liang Chen, Chenyun Yu, Lei Cheng, ChengXiang Zhuo, Bo Hu, Zang Li
Large Language Model Natural Language Processing Context Learning Human Attention LLM Based Text to SQL Microbial Decomposition Real World Office Workflow Context Learning Ability

February 10, 2024

In-Context Data Distillation with TabPFN
Junwei Ma, Valentin Thomas, Guangwei Yu, Anthony Caterini
Tabular Datasets Context Learning Ability Task Specific Training

November 15, 2023

Auto-ICL: In-Context Learning without Human Supervision
Jinghan Yang, Shuming Ma, Furu Wei
Context Learning Context Learning Ability

October 26, 2023

In-Context Ability Transfer for Question Decomposition in Complex QA
Venktesh V, Sourangshu Bhattacharya, Avishek Anand
Complex Question Question Decomposition Context Learning Ability Complex Question Answering

October 18, 2023

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations
Arkil Patel, Satwik Bhattamishra, Siva Reddy, Dzmitry Bahdanau
Context Learning Long Context Natural Language Description Context Learning Ability Ambiguous Interpretation

October 12, 2023

Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
Ivan Lee, Nan Jiang, Taylor Berg-Kirkpatrick
Context Learning Human Attention Context Example Human Relationship Model Architecture Context Learning Ability Context Learner

August 3, 2023

Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Zheyu Zhang, Han Yang, Bolei Ma, David Rügamer, Ercong Nie
Language Model Language Understanding LLM Generated Compact Model Context Learning Ability Cognitive Reasoning

July 15, 2023

SINC: Self-Supervised In-Context Learning for Vision-Language Tasks
Yi-Syuan Chen, Yun-Zhu Song, Cheng Yu Yeo, Bei Liu, Jianlong Fu, Hong-Han Shuai
Self Supervised Context Learning Vision Language Task Large Pre Trained Transformer Context Learning Ability

July 3, 2023

Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking
Qinyong Wang, Zhenxiang Gao, Rong Xu
Large Language Model Biomedical Concept Biomedical Entity Linking Context Learning Ability Chemical Named Entity Recognition

June 2, 2023

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
Masoud Monajatipoor, Liunian Harold Li, Mozhdeh Rouhsedaghat, Lin F. Yang, Kai-Wei Chang
Language Model Full Model Context Learning Pre Trained Vision Language Model Vision Language Task Critique Ability Different Context Context Learning Ability Context Learning Capability

May 23, 2023

Concept-aware Training Improves In-context Learning Ability of Language Models
Michal Štefánik, Marek Kadlčík
Language Model Context Learning Context Information Context Learning Ability Context Learner

May 18, 2023

Learning In-context Learning for Named Entity Recognition
Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, Le Sun
Context Learning Entity Recognition Entity Extraction Context Learning Ability Shot NER Class Incremental NER

May 17, 2023

Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries
Jiageng Wu, Xian Wu, Zhaopeng Qiu, Minghui Li, Yingying Zhang, Yefeng Zheng, Changzheng Yuan, Jie Yang
Large Language Model Medical Question Answering Linguistic Landscape Context Learning Ability English Centric

December 3, 2022

Can In-context Learners Learn a Reasoning Concept from Demonstrations?
Michal Štefánik, Marek Kadlčík
Language Model Noisy Demonstration Reasoning System Context Demonstration Context Learning Ability Context Learner

September 24, 2022

In-context Learning and Induction Heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah
Context Learning Large Transformer Model Head Transformer Context Learning Ability

April 28, 2022

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung
Context Learning Shot Learning Mixed Effect Large Scale Language Model Training Corpus Shot in Context Learning Pre Training Corpus Context Learning Ability

Context Learning Ability

Papers

Leveraging Large Language Models for Wireless Symbol Detection via In-Context Learning

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification

Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm

In-Context Data Distillation with TabPFN

Auto-ICL: In-Context Learning without Human Supervision

In-Context Ability Transfer for Question Decomposition in Complex QA

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

SINC: Self-Supervised In-Context Learning for Vision-Language Tasks

Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Concept-aware Training Improves In-context Learning Ability of Language Models

Learning In-context Learning for Named Entity Recognition

Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries

Can In-context Learners Learn a Reasoning Concept from Demonstrations?

In-context Learning and Induction Heads

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model