SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0
Organizations: Department of Computer Science, Texas A&M University–Corpus Christi, Corpus Christi, TX, USA. · School of Engineering and Computing, Christopher Newport University, Newport News, VA, USA.
Abstract
Closed-context question answering (QA) requires a system to generate an answer strictly supported by a supplied passage, or to abstain when the passage lacks adequate evidence. SQuAD 2.0 is a widely used benchmark for this setting, scored by Exact Match (EM) i.e., character-level agreement with human reference spans after lexical normalization. For generative large language models (LLMs), however, an EM failure may reflect a valid paraphrase or an alternative passage-supported answer rather than a hallucination. The best prior leaderboard EM is approximately 91%, but evaluating generative models requires separating lexical divergence from factual error. We introduce SAGE-QA (Semantic-Aware, Grounded Evaluation for Question Answering), a multi-tiered framework combining reference-based Semantic Equivalence Scoring (SES) and passage-based Re-Validation Scoring (RVS). On a fixed set of 10,000 SQuAD 2.0 validation examples, we compute EM and decision-matrix F1 from TP, TN, FP, and FN counts for zero-shot closed-weight frontier models and for open-weight models, untuned and fine-tuned. Zero-shot, Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, outperforming untuned Llama 3 8B (56.88), gpt-oss-20b (66.28), and Llama 3.3 70B (71.39). Parameter-efficient fine-tuning adds 21.19 to 34.06 EM points, raising these models to 90.94, 87.47, and 94.24 EM, respectively. Auditing the frontier models with SAGE-QA and an external LLM judge yields dataset-wide SES scores of 92.81 for Claude Opus 4.6 and 91.67 for Gemini 3.1 Pro, and hybrid RVS scores, which combine benchmark credit with passage-based judge assessments, of 97.96 and 98.02. Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM.
Figures & tables
| System | EM | |
|---|---|---|
| Human Performance | 86.831 | 89.452 |
| IE-Net (Discriminative Ensemble) | 90.939 | 93.214 |
| Stage | Target Pool / Metric | Operational Definition and Inputs |
|---|---|---|
| 1. Generation | All items | Model generates a raw answer field from , using [] to indicate abstention. |
| 2. Interpretation and Lexical Match | , | Determine from the raw answer and retain interpreted text . Normalize only for EM; assign outcomes using , and split using . |
| 3a. SES Judge | External judge scores semantic equivalence using only . | |
| 3b. RVS Judge | Same external judge scores passage support (and binary acceptance at ) using only . | |
| 4. Aggregation | Report eligible case means and hybrid dataset-wide scores combining benchmark-based credit for and with judge-assigned credit, without double-counting overlapping cases. |
| LoRA | Learning | Batch | Max Seq. | ||
|---|---|---|---|---|---|
| Model | Rank ( ) | Rate | Size | Len. | |
| Llama 3 8B | 128 | 32 | 32 | 512 | |
| gpt-oss-20b | 128 | 32 | 64 | 512 | |
| Llama 3.3 70B | 32 | 64 | 1,024 |
| Cohort | Model | TP | FN ( ) | FP | TN | FP Rate (%) | FN Rate (%) | EM | F1 |
| Zero-Shot (Frontier) | Claude Opus 4.6 | 3,937 | 1,053 (165 / 888) | 426 | 4,584 | 8.50 | 21.10 | 85.21 | 84.19 |
| Zero-Shot (Frontier) | Gemini 3.1 Pro | 4,220 | 770 (171 / 599) | 564 | 4,446 | 11.26 | 15.43 | 86.66 | 86.35 |
| Zero-Shot (Open-Weight) | Llama 3 8B (Untuned) | 3,236 | 1,754 (107 / 1,647) | 2,558 | 2,452 | 51.06 | 35.15 | 56.88 | 60.01 |
| Zero-Shot (Open-Weight) | Llama 3.3 70B (Untuned) | 3,576 | 1,414 (120 / 1,294) | 1,447 | 3,563 | 28.88 | 28.34 | 71.39 | 71.43 |
| Zero-Shot (Open-Weight) | gpt-oss-20b (Untuned) | 3,610 | 1,380 (112 / 1,268) | 1,992 | 3,018 | 39.76 | 27.66 | 66.28 | 68.16 |
| Fine-Tuned (Open-Weight) | Llama 3 8B (Fine-Tuned) | 4,681 | 309 (—) | 597 | 4,413 | 11.92 | 6.19 | 90.94 | 91.18 |
| Claude | Gemini | |
| Diagnostic Metric | Opus 4.6 | 3.1 Pro |
| Baseline EM (Table IV ) | 85.21 | 86.66 |
| Eligible SES Pool | 888 | 599 |
| (Mean, 0–100) | 85.56 | 83.68 |
| (Graded Dataset-Wide, 0–100) | 92.81 | 91.67 |
| Eligible RVS Pool | 1,314 | 1,163 |
| Evaluated Pool ( ) | |||||
|---|---|---|---|---|---|
| Claude Opus 4.6 | |||||
| SES on ( ) | 96 | 219 | 493 | 50 | 30 |
| RVS on ( ) | 373 | 12 | 18 | 3 | 20 |
| RVS on ( ) | 825 | 21 | 30 | 6 | 6 |
| Gemini 3.1 Pro | |||||
| Claude | Gemini | |
|---|---|---|
| Diagnostic Category | Opus 4.6 | 3.1 Pro |
| Paired Decisions on Answered FN Cases ( ) | ||
| Accepted by Both ( ) | 310 | 242 |
| Accepted by SES Only ( ) | 5 | 5 |
| Accepted by RVS Only ( ) | 536 | 321 |
| Accepted by Neither ( ) | 37 | 31 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,539 papers