Different Context

In-context learning (ICL) explores how large language models (LLMs), particularly transformer-based architectures, can solve new tasks by processing a few demonstration examples alongside a query, without explicit parameter updates. Current research focuses on understanding ICL's mechanisms, improving its effectiveness through prompt engineering and data selection strategies, and applying it to diverse domains like robotics, PDE solving, and deepfake detection. This research is significant because it offers a more efficient and adaptable alternative to traditional fine-tuning, potentially impacting various fields by enabling faster model adaptation and reducing the need for extensive labeled datasets.

Papers

November 11, 2024

Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks
Madeline Brumley, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, Usman Anwar
Large Language Model Inherent Interpretability Constructive Approach Different Context Interpretability Research Learning Task Steering Vector Context Vector Bottom Up Framework

November 6, 2024

Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks
Ryan Campbell, Nelson Lojo, Kesava Viswanadha, Christoffer Grondal Tryggestad, Derrick Han Sun, Sriteja Vijapurapu, August Rolfsen, Anant Sahai
Full Model Context Learning Efficient Hybrid Different Context Sequence Model Efficient Architecture Task Learning

November 5, 2024

Mixtures of In-Context Learners
Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, Pasquale Minervini
Context Learning Mixture Component Noisy Demonstration Different Context Transformer Based LLM Context Learner

November 3, 2024

Robust Neural Processes for Noisy Data
Chen Shapira, Dan Rosenbaum
Context Learning Noisy Data Different Context Neural Process Neural Network Robustness

October 30, 2024

Toward Understanding In-context vs. In-weight Learning
Bryan Chan, Xinyi Chen, András György, Dale Schuurmans
Context Learning Context Aware Different Context Weight Freezing Distributional Property

October 22, 2024

Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods
Tsachi Blau, Moshe Kimhi, Yonatan Belinkov, Alexander Bronstein, Chaim Baskin
Large Language Model Context Learning Shot Learning Prompt Tuning Different Context Token Embeddings Adversarial Method

October 17, 2024

On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery
Renpu Liu, Ruida Zhou, Cong Shen, Jing Yang
Transformer Megatron Decepticons Context Learning Gradient Descent Different Context Learned Optimizers Sparse Recovery Standard Gradient Descent

October 16, 2024

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
Yunqiu Xu, Linchao Zhu, Yi Yang
New Benchmark Multimodal Large Language Model Visual Grounding New Era Different Context MT Bench

October 15, 2024

October 13, 2024

Can In-context Learning Really Generalize to Out-of-distribution Tasks?
Qixun Wang, Yifei Wang, Yisen Wang, Xianghua Ying
Context Learning Distribution Shift Context Example Different Context Learning Task

October 10, 2024

Metalic: Meta-Learning In-Context with Protein Language Models
Jacob Beck, Shikha Surana, Manus McAuliffe, Oliver Bent, Thomas D. Barrett, Juan Jose Garau Luis, Paul Duckworth
Protein Sequence Different Context Protein Language Model Protein Design Fitness Approximation Heavy Metal

October 9, 2024

SEGMENT+: Long Text Processing with Short-Context Language Models
Wei Shi, Shuang Li, Kerun Yu, Jinglei Chen, Zujie Liang, Xinhui Wu, Yuxi Qian, Feng Wei, Bo Zheng, Jiaqing Liang, Jiangjie Chen, Yanghua Xiao
Language Model Different Context Well Defined Segment Long Document Structured Document Long Input

October 8, 2024

October 7, 2024

Different Context

Papers

Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks

Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks

Mixtures of In-Context Learners

Robust Neural Processes for Noisy Data

Toward Understanding In-context vs. In-weight Learning

Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods

On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning

On the Training Convergence of Transformers for In-Context Classification

State-space models can learn in-context by gradient descent

Towards More Effective Table-to-Text Generation: Assessing In-Context Learning and Self-Evaluation with Open-Source Models

Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent

Can In-context Learning Really Generalize to Out-of-distribution Tasks?

Metalic: Meta-Learning In-Context with Protein Language Models

SEGMENT+: Long Text Processing with Short-Context Language Models

Vector-ICL: In-context Learning with Continuous Vector Representations

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

Density estimation with LLMs: a geometric investigation of in-context learning trajectories

Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning