In-Context Example Selection

Overview

The field studies which labeled examples, demonstrations, or other context to place in a model’s prompt so it can generalize to a new query. Common approaches retrieve semantically similar examples, while others optimize for diversity and latent coverage, task-specific utility, decision-boundary cases, or signals from model activations; learned ranking, bandit, and reinforcement-learning methods also refine selections. Methods are evaluated across settings such as classification, reasoning, translation, multimodal tasks, and tabular prediction, often measuring accuracy alongside prompt length, latency, or robustness to noise. A central tension is that similarity is cheap but can miss useful contrasts, while more tailored selection can improve results yet may be costly or fail to transfer across models and tasks.

Latest papers

32 papers3 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Feb 12, 2026

TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction

Tabular prediction can benefit from in-table rows as few-shot evidence, yet existing tabular models typically perform instance-wise inference and LLM-based prompting is often brittle. Models do not consistently leverage relevant rows, and noisy context can degrade performance. To address this challenge, we propose TabSieve, a select-then-predict framework that makes evidence usage explicit and auditable. Given a table and a query row, TabSieve first selects a small set of informative rows as evidence and then predicts the missing target conditioned on the selected evidence. To enable this capability, we construct TabSieve-SFT-40K by synthesizing high-quality reasoning trajectories from 331 real tables using a strong teacher model with strict filtering. Furthermore, we introduce TAB-GRPO, a reinforcement learning recipe that jointly optimizes evidence selection and prediction correctness with separate rewards, and stabilizes mixed regression and classification training via dynamic task-advantage balancing. Experiments on a held-out benchmark of 75 classification and 52 regression tables show that TabSieve consistently improves performance across shot budgets, with average gains of 2.92% on classification and 4.45% on regression over the second-best baseline. Further analysis indicates that TabSieve concentrates more attention on the selected evidence, which improves robustness to noisy context.
Aug 20, 2025

Beyond Semantic Similarity: Reducing Unnecessary API Calls via Behavior-Aligned Retriever

Tool-augmented LLMs invoke external functions to extend their capabilities, but errors in the invocation decision, such as calling a tool when none is needed or omitting a needed call, can produce unreliable outputs and unnecessary cost. A lightweight remedy is to prepend retrieved examples so LLMs decide tool use in context. However, existing retrievers rank examples by semantic similarity alone. Lexically close or semantically close queries can require opposite behavior, so the retrieved examples may be behaviorally inconsistent and silently mislead the model. We propose Behavior Aligned Retrieval (BAR), a backbone-agnostic training recipe that teaches a dense retriever a behavior-aware similarity, keeping semantically related candidates close only when their tool-use behavior is compatible. BAR does not predict invocation labels; instead, it ranks demonstrations while leaving the final tool-use decision to the LLM. Applied to multiple retrieval backbones, including BERT, Contriever, and Qwen-based representation backbone, BAR consistently improves invocation reliability and reduces unnecessary API calls across 14 LLMs and 3 benchmarks.