Retrieval-Augmented ICL

ICL: In-Context Learning

Retrieval-augmented in-context learning studies how to select and present examples or other evidence so a pretrained model can perform a task without changing its weights. Methods range from nearest-neighbor search with text or multimodal embeddings to task-aware retrievers trained with model feedback, structured context selection, and iterative agents that identify information gaps. These approaches are used in settings such as question answering, classification, reasoning, medical image interpretation, and robotics, with benchmarks often comparing retrieval against random or fixed demonstrations and against fine-tuning. A central open tension is that similarity does not always predict usefulness: retrieval must balance task relevance, context budget, evidence format, inference cost, and risks such as privacy leakage or negative transfer.

Latest papers

45 papers8 papers in the last four weeks, down 27% on the four weeks before. 0.0% of all new papers.

In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
May 25, 2026Mingchen Li, Jiatan Huang, Chuxu Zhang +2Retrieval-Augmented Generation Test-Time Adaptation
University of Massachusetts, Amherst · University of Connecticut · Emory University
Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering
May 5, 2026Tejas Kulkarni, Antti Koskela, Laith ZumotIn-Context Retrieval Membership Inference Attacks
Nokia Bell Labs · Nokia
Job Skill Extraction via LLM-Centric Multi-Module Framework
Apr 23, 2026Guojing Li, Zichuan Fu, Junyi Li +8Document Information Extraction LLM Information Extraction
City University of Hong Kong · Renmin University of China · 1City University of Hong Kong. +1
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
Apr 21, 2026Yuqi Zhang, Di ZhangRetrieval-Augmented Generation Self-Attention
Xi’an Jiaotong-Liverpool University
Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning
Apr 7, 2026Le Liu, Zhiming Li, Jianzhi Yan +7In-Context Learning Cross-Domain Transfer Learning
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China
Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval
Apr 7, 2026Jianzhi Yan, Zhiming Li, Le Liu +6Cross-Domain Transfer Learning Retrieval-Augmented ICL
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China
Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning
Apr 4, 2026Tianci Luo, Haohao Pan, Jinpeng Wang +5In-Context Learning Retrieval-Augmented ICL
Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Computer Science and Engineering, Northeastern University · Harbin Institute of Technology, Shenzhen
Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts
Apr 3, 2026Jinsook Lee, Kirk Vanacore, Zhuqian Zhou +2LLM-Assisted Annotation Retrieval-Augmented ICL
Cornell University Ithaca, NY, 14850, USA
Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints
Apr 2, 2026Minh-Khoi Pham, Thang-Long Nguyen Ho, Thao Thi Phuong Dao +8Tabular Foundation Models Retrieval-Augmented ICL
School of Computing, Dublin City University, Dublin, Ireland · ADAPT Centre, Dublin City University, Dublin, Ireland · University of Science, Vietnam National University, Ho Chi Minh City, Vietnam +5
Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis
Mar 29, 2026Wenkai Zhao, Zipei Wang, Mengjie Fang +3Multimodal ICL Medical VLMs
CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences · School of Software, North University of China
Retrieving Counterfactuals Improves Visual In-Context Learning
Mar 17, 2026Guangzhi Xiong, Sanchit Sinha, Zhenghao He +1Multimodal ICL Retrieval-Augmented ICL
University of Virginia
Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation
Mar 10, 2026Eeham Khan, Luis Rodriguez, Marc QueudotRAG Citation Faithfulness Retrieval-Augmented Generation
Concordia University, Montréal, QC, Canada · Centre de recherche informatique de Montréal (CRIM), Montréal, QC, Canada
M $^2$ : Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval
Feb 28, 2026Dawei Yan, Haokui Zhang, Guangda Huzhang +8Long-Horizon LLM Agents Web Agents
Northwestern Polytechnical University, Xi’an, China · AI Business, Alibaba Group, Hangzhou, China · Xi’an University of Architecture and Technology, Xi’an, China +1
Evolutionary Context Search for Automated Skill Acquisition
Feb 18, 2026Qi Sun, Stefan Nielsen, Rio Yokota +1Retrieval-Augmented ICL LLM Agent Skill Learning
Sakana AI, Japan · Institute of Science Tokyo, Japan
LLM Program Optimization via Retrieval Augmented Search
Jan 31, 2025Sagnik Anupam, Alexander Shypula, Osbert BastaniLarge Language Model-Guided Optimization In-Context Learning
University of Pennsylvania

May 25, 2026J/K move · Enter open · S save

In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective

Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao+1

University of Massachusetts, Amherst · University of Connecticut · Emory University

Retrieval-Augmented Generation Test-Time Adaptation Adaptive RAG In-Context Learning Retrieval-Augmented ICL

Open paper AskPDF arXiv

In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost.