cs.AINov 12, 2026

SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0

Authors: Chandra N Sekharan, Vishakan Umapathy, Abhishek Phadke

Organizations: Department of Computer Science, Texas A&M University–Corpus Christi, Corpus Christi, TX, USA. · School of Engineering and Computing, Christopher Newport University, Newport News, VA, USA.

Abstract

Closed-context question answering (QA) requires a system to generate an answer strictly supported by a supplied passage, or to abstain when the passage lacks adequate evidence. SQuAD 2.0 is a widely used benchmark for this setting, scored by Exact Match (EM) i.e., character-level agreement with human reference spans after lexical normalization. For generative large language models (LLMs), however, an EM failure may reflect a valid paraphrase or an alternative passage-supported answer rather than a hallucination. The best prior leaderboard EM is approximately 91%, but evaluating generative models requires separating lexical divergence from factual error. We introduce SAGE-QA (Semantic-Aware, Grounded Evaluation for Question Answering), a multi-tiered framework combining reference-based Semantic Equivalence Scoring (SES) and passage-based Re-Validation Scoring (RVS). On a fixed set of 10,000 SQuAD 2.0 validation examples, we compute EM and decision-matrix F1 from TP, TN, FP, and FN counts for zero-shot closed-weight frontier models and for open-weight models, untuned and fine-tuned. Zero-shot, Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, outperforming untuned Llama 3 8B (56.88), gpt-oss-20b (66.28), and Llama 3.3 70B (71.39). Parameter-efficient fine-tuning adds 21.19 to 34.06 EM points, raising these models to 90.94, 87.47, and 94.24 EM, respectively. Auditing the frontier models with SAGE-QA and an external LLM judge yields dataset-wide SES scores of 92.81 for Claude Opus 4.6 and 91.67 for Gemini 3.1 Pro, and hybrid RVS scores, which combine benchmark credit with passage-based judge assessments, of 97.96 and 98.02. Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM.

Figures & tables

Explore similar work

CardsList
  1. How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

    Sep 1, 2026Elitsa Yotkova, Violeta Kastreva, Petar Velkov +4LLM Answer VerificationLanguage Model Generation Evaluation

  2. SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

    Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM EvaluationLanguage Model Generation Evaluation

  3. Semantic Uncertainty Quantification Needs Factual Equivalence

    Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language ModelsFactual Consistency Evaluation