Source Scope Items Clusters [0pt][0pt] Main split FRED Time 200 43 Location 200 44 [1pt/1.5pt] SituatedQA Time 183 137 Location 184 116 [1pt/1.5pt] AmbigDocs Entity 192 192 [1pt/1.5pt] Total 959 489 [0pt][0pt] Supplement CondAmbigQA Condition 25 20
3.3 Construction
We build items from all sources under the same construction guideline (Appendix A.1 ). Each item is built in three steps: writing the question, creating the contexts, and selecting the answer options.
Questions. Each item starts from a fact pair: the gold fact ⟨s,y∗⟩ and the lure fact ⟨s′,ℓ⟩ . We verify that both facts are supported by the source in their subject, attribute, scope, and value, and that y∗ and ℓ remain distinct after normalizing aliases and formats. We then write a question that specifies the target scope s without revealing either value. Contexts. G and S express the gold and lure facts, respectively, and state all relevant scope qualifiers explicitly. Depending on the source, they are generated from FRED records with fixed templates, taken as contiguous sentences from source documents, or written as faithful restatements of source content, with GPT-6 Astra ( OpenAI, 2026 ) assisting in wording. F is derived from G by replacing every expression of the gold answer, including aliases, with the lure, leaving all other text unchanged. G and F therefore differ only in the asserted value. We exclude items in which this substitution leaves a gold mention, produces ungrammatical text, or creates unintended contradictions (Appendix A.1 ).
Answer options. Each item has five options: the gold answer y∗ , the lure ℓ , two distractors d1 and d2 , and Insufficient information ( ⊥ ), which is always option E. Distractors are values of the same attribute at other documented scopes, preferably from the same upstream question. They must be wrong for the target scope, distinct from the other options after normalization, similar to the gold answer in type and precision, and absent from the question and contexts. Because models are sensitive to option positions ( Zheng et al., 2024 ) , we balance the gold and lure positions across A–D and the 12 ordered position pairs within each source–scope group of the main split, with counts differing by at most one item (Appendix B.1 ). 3.4 Quality Control
We apply three layers of quality control: automated checks, model screening, and a human audit.
Automated checks. Scripts verify that each item is well-formed, that its options remain distinct after normalization, and that no distractor appears in the question or contexts. They also check that every fact is traceable to its source, that F differs from G only in the edited answer spans, and that context hashes and release checksums match. All released items pass these checks.
Semantic screening. Two models outside the evaluation panel screen every item with a shared prompt: GPT-6 Astra ( OpenAI, 2026 ) in a fresh session without construction history, and Claude Opus 5.5 ( Anthropic, 2026 ) , which was not used in construction. The prompt asks whether each fact is source-supported, whether G and F match the target scope while S differs in exactly one dimension, whether the gold answer is unique, whether F reads fluently, and whether the distractors are valid (Appendix B.1 ). Either model flags at least one criterion for 266 items. A human reviewer inspects each flagged item, excludes 64 of them, and records the reason for every decision.
Human audit. To check whether the final items satisfy the intended design, we audit a stratified sample of 150 main items: 15 per FRED scope, 35 per SituatedQA scope, and 50 from AmbigDocs. Each item is independently reviewed by two of six annotators, with 50 items per annotator, using the criteria in Appendix B.2 . Both annotators judge 149 of the 150 items (99.3%) to satisfy all criteria. All model evaluations use the same frozen version of MisleadQA (Appendix B.1 ).
4 Experiments
We evaluate how true but out-of-scope context (S) and false but in-scope context (F) affect model answers, when presented alone and together with gold context (G), on questions each model already answers correctly without any context. This section describes the evaluation setup (§ 4.1 ), the metrics (§ 4.2 ), and statistical inference (§ 4.3 ).