cs.SEAug 14, 2026

The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context

Authors: Alexander Adkins, Teimuraz Trapaidze

Organizations: Make Games Initiative

Abstract

Retrieval components for code assistants are tuned against retrieval metrics: a configuration that raises recall@k is adopted, and downstream task success is assumed to follow. We report a controlled case study in code repair, not a new phenomenon but a deployed-flag, execution-graded instance of the known relevance-diversity and objective-mismatch tradeoff (Levy et al., 2025). On SWE-bench Verified we inject a retriever's hits as a fixed 12-slot context pack with no search tools and toggle one flag (one-chunk-per-file deduplication) on an otherwise identical stack. The flag is the higher-recall configuration (gold file present in 0.878 of served packs against 0.806 disabled), yet disabling it, trading file breadth for within-file depth, raises the single-shot resolve rate: gpt-5.6-sol +7.6pp (39.2% to 46.8%, n=500, McNemar exact p=0.0003), and a pre-registered open-weights replication any reviewer can re-run (Qwen3.6-27B, +3.6pp, n=499, p=0.0133); both survive repository-clustered inference. The gain tracks within-file anchor dose, and a random-chunk control refutes an argmax-selection artifact. We map where it holds: it reverses on a lexical BM25 retriever (-3.2pp, significant cross-paradigm interaction), is not detected under unrestricted-Read agents (a powered null), and across four languages (SWE-PolyBench, N=617) is positive but not significant (+2.6pp, p=0.056), a mapped boundary rather than a confirmed extension. Operationally, at a tight fixed budget: do not hard-deduplicate by file, and A/B packing policies against the task, not the metric the flag was tuned to.

Explore similar work

CardsList
  1. Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

    Jul 27, 2026Bowen Qin, Yi XieSoftware Engineering AgentsAI Agent Benchmarks

  2. ContextSniper: AntTrail's Token-Efficient Code Memory for Repository-Level Program Repair

    Jul 2, 2026Chiwang Luk, Matin Mohammad Najafi, Zhifeng Jia +6Automated Program RepairAI Coding Agents

  3. CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

    Aug 6, 2026Wuya Chen, Yihao yang, Yang Cao +1AI Coding AgentsLLM Agent Workflow Optimization