cs.CLFeb 27, 2026

Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam

Authors: Michael Majurski, Cynthia Matuszek

Organizations: National Institute of Standards and Technology · University of Maryland Baltimore County

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

How carefully and unambiguously a question is phrased has a profound impact on the quality of the response, for Language Models (LMs) as well as people. While model capabilities continue to advance, the interplay between grounding context and query formulation remains under-explored. This work investigates how the quality of background grounding information in a model's context window affects accuracy. We find that combining well-grounded dynamic context construction (i.e, RAG) with query rewriting reduces question ambiguity, resulting in significant accuracy gains. Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark improvements without changing the answer itself, even compared to prepending that context before the question. Using gpt-oss-20b to rewrite a subset of Humanity's Last Exam using answer-free grounding context improves gpt-5-mini accuracy from 0.14 to 0.37. We demonstrate that this accuracy improvement cannot be fully recovered just through prompting at inference time; rather, distinct rewriting and answering phases are required. Code and data are available at https://github.com/mmajurski/lm-rewrite-uplift

Figures & tables

Figure 1: When RAG systems surface relevant information, LM performance can be enhanced by rewriting the initial query using context —added information that, without providing the answer, gives relevant background knowledge and direction.
Figure 2: Rewriting questions using answer-free grounding context yields significant accuracy improvement (distance above line) over the original questions, as evaluated on a subset of Humanity’s Last Exam (HLE). See Figure 4 for a complete legend.
Figure 3: The quality of context presented to an LM has an impact on question-answering performance. Accuracy is highest when RAG systems surface context containing the answer ( cyan ), but when the question is presented without context ( orange ) or the surfaced information does not contain the answer ( green ), performance (without query rewriting) suffers. The act of interpretation during question rewriting ( pink ) produces an accuracy improvement beyond just prepending the Question + AnswerFree Context information used to rewrite before the question; despite not including the AFC when asking an LM the rewritten question.
DatasetRelease Date
Public Datasets 
Humanity’s Last Exam [ 18 ]2025
Squadv2 [ 19 ]2018
HotpotQA [ 32 ]2018
TrivaQA-web [ 11 ]2017
NaturalQuestionsShort [ 13 ]2019
Table 1: Evaluation Datasets and Publication Dates
ModelKnowledge-CutoffPublic-Release
gpt-5Sep 2024Aug 2025
gpt-5-miniMay 2024Aug 2025
gpt-5-nanoMay 2024Aug 2025
gpt-oss-20bJun 2024Aug 2025
gpt-oss-120bJun 2024Aug 2025
gemma-3-1b-itAug 2024Mar 2025
Table 2: Evaluation Models Knowledge Cutoff
Figure 4: Performance of various LMs with and without answer-containing context and answer-free context. The x -axis shows the original question benchmark accuracy while the y -axis shows the rewritten question benchmark accuracy; distance above the line conveys improvement. Colors represent the different models tested ( table 2 ) and shapes represent the dataset ( table 1 ). (top left) Answers are present in the context (baseline) : Benchmark performance significantly improves with the addition of context that contains the answer. (top right) Answers not present in the context : Simply providing relevant answer free context without question rewriting does not improve performance. (bottom left) Questions rewritten using answer-free context (our approach) : Improvement in accuracy caused by rewriting the question only using AFC where during benchmarking only the rewritten question is presented to the LM ( AFC is withheld).
Figure 5: Per-dataset per-model difference in benchmark accuracy between the rewritten questions and the original questions with associated answer-free context. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1346.
Figure 6: Per-dataset difference in benchmark accuracy between the rewritten questions without AFC and the original questions with AFC . The violin plot distribution highlights the range of accuracy deltas over all models for each dataset evaluated. Benchmark accuracy improved by an average of 0.1346.
Figure 7: Per-dataset improvement in benchmark accuracy from Rewrite_Q+AFC compared to the Orig_Q+AFC during benchmark evaluation. Benchmark accuracy improved by an average of 0.0875.
Figure 8: Accuracy improvement on the x -axis vs. improvement in cosine similarity between the question and context for rewritten questions on the y -axis. x -axis: delta benchmark accuracy Rewrite_Q −- Orig_Q ; larger values indicate that the rewritten question improved benchmark accuracy. y -axis: increase in cosine similarity between the question and context due to rewriting; larger values indicate better alignment between Rewrite_Q and AFC compared to Orig_Q and AFC . The fact that most points are in the upper right quadrant demonstrates that improvements in embedding alignment between the question and grounding context correlate with improved benchmark accuracy.
Figure 9: Per-dataset improvement in benchmark accuracy from performing an in situ rewrite of the question using answer-free context during benchmark evaluation. This combines the rewrite-then-answer method into a single operation. The prior accuracy improvement disappears, highlighting the impact of task separation between the rewrite and answer phases.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Per-dataset per-model difference in benchmark accuracy between the rewritten question and the original question. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1303.
Figure 11: Per-dataset per-model difference in benchmark accuracy between the rewritten question and the original question. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1303.
Figure 12: Per-model difference in benchmark accuracy between the rewritten questions and the original questions with associated answer-free context. The violin plot distribution highlights the range of accuracy deltas over all datasets for each model evaluated. Benchmark accuracy improved by an average of 0.1346.
Figure 13: Per-dataset per-model difference in benchmark accuracy between the rewritten questions and the original questions with associated answer-free context. The violin plot distribution highlights the range of accuracy deltas over all models for each dataset evaluated. Benchmark accuracy improved by an average of 0.1346.
Figure 14: Per-dataset improvement in benchmark accuracy from Rewrite_Q with AFC compared to the Orig_Q with AFC during benchmark evaluation. Benchmark accuracy improved by an average of 0.0875.
More in this topic

Query Reformulation

Query reformulation transforms user questions into retrieval-ready queries to improve search, evidence coverage, and downstream answers.

See all 42 papers

Explore similar work

CardsList

  1. Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

    May 24, 2026Jinyan Su, Claire CardieLLM EvaluationClarification Question Generation

    1Cornell University

  2. Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction

    Oct 25, 2026Zhe Xu, Tianhao Liao, Zhenqing Ling +3Long-Context RetrievalInformation Retrieval

    Sun Yat-sen University · Alibaba Group

  3. Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem

    Feb 25, 2026Heejin JoStructured Reasoning in Language ModelsPrompt Engineering

    Independent Researcher