cs.IRNov 2, 2026

Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions

Authors: S. M. Mahbub Uddin Ahmed, Abdullah Al Musaddiq

Organizations: United International University Dhaka, Bangladesh · Islamic University of Technology Gazipur, Bangladesh

Abstract

People often attach instructions to their search queries, asking for pages written for beginners, documents in a certain language or official sources only. Most rerankers were trained to judge topical relevance and treat such instructions as a few extra words. On the InfoSearch benchmark, even GPT-4o used as a list-wise reranker reaches an average WISE (Weighted Instruction Sensitivity Evaluation) of only 33.5. We present DMReranker (Decision-Model Reranker), a zero-shot reranker that splits an instructed query into a topic and a short list of typed clauses. A small judge model answers one narrow question per clause, and the answer probabilities are multiplied into a single score. Nothing is trained, and the configuration was frozen before the test runs. Reranking the same top-100 candidates as the published baselines, DMReranker reaches a macro WISE of 50.9 over all six InfoSearch dimensions while keeping original-mode nDCG@10 at 82.9. It trails GPT-4o on the stricter SICR (Strict Instruction Compliance Ratio) metric, 19.2 against 32.1. We show that this gap follows from how a product of probabilities behaves when SICR compares raw scores across runs. Computed from ranks, our SICR is 67.0. On the same judge, splitting the instruction raises WISE from 32.9 to 47.6 over a single combined question, while raw-score SICR falls. Scoring all six dimensions cost about $10 at list price.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Re3^3Rank: Listwise Reranking with Reasoning-Conditioned Retrieval Heads

    Nov 2, 2026Ruiyi Yan, Zhuoyuan Mao, Yugo Murawaki +1Learning to RankLLM Reranking

  2. On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

    Sep 1, 2026Vignesh Prabhakar, Jialing Pan, Anil Babu AnkisettipalliReinforcement LearningLearning to Rank

  3. Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

    Oct 26, 2026Qiaomu Li, Ying XieLLM RerankingRAG Evaluation