Joint Constraint Aggregation for Qualifier-Sensitive Retrieval-Augmented QA

Jennifer Hsia; Zachary C. Lipton; Aarti Singh

cs.CLOct 19, 2026

Joint Constraint Aggregation for Qualifier-Sensitive Retrieval-Augmented QA

Authors: Jennifer Hsia, Zachary C. Lipton, Aarti Singh

Organizations: Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA

Abstract

Retrieval-augmented QA can fail even when the correct answer is in the retrieved evidence. This can happen when a competing answer satisfies some, but not all, of the question's requirements and is mentioned frequently in the retrieved documents, which can make it appear to be the best-supported answer. To test whether this retrieval skew actually affects the decision, we manipulate it directly: increasing the prevalence of passages that favor the competing answer makes the model more likely to select it. This points to a decision problem that candidate-level support alone cannot resolve. We therefore introduce Joint Constraint Aggregation (JCA), which compares candidates against the same question-derived requirements using the retrieved evidence. Across three evaluation settings and three instruction-tuned backbones, JCA improves overall accuracy, and its advantage grows when retrieval favors a competing answer. Together, these results suggest that when partial matches and retrieval prominence pull models toward a competing answer, comparing candidates against the question's full requirements can make selection more reliable.

Figures & tables

Figure 1: Illustration of qualifier-sensitive answer selection and Joint Constraint Aggregation. Retrieved webpages mention a plausible near-miss, Ariana Grande in the film adaptation of Wicked , more frequently than the qualifier-satisfying answer, Kristin Chenoweth in the original Broadway production. A baseline selector chooses the near-miss despite the Broadway qualifier. JCA represents the question requirements as shared constraints, checks each candidate against those constraints using retrieved evidence, and jointly compares the resulting reports.

	AmbigQA		AmbigQA-Extracted		SituatedQA
	( $n$ = 796 / 294)		( $n$ = 548 / 198)		( $n$ = 674 / 238)
Method	Full	Hard	Full	Hard	Full	Hard
Direct	84.2	75.9	84.5	79.3	62.9	47.5
CoT	87.4	82.3	89.6	85.9	74.2	64.3
Compare	82.9	71.1	81.0	68.7	65.9	47.1
NLI	78.9	66.7	83.0	77.3	59.6	45.4

Table 1: Accuracy (%) on the full evaluation set and hard slice across three evaluation conditions using Llama4-Scout-17B. Hard : mention gap

<0

n

gives full / hard instances. JCA achieves the highest accuracy in every column.

Backbone	AmbigQA	AmbigQA-Extracted	SituatedQA
Qwen3-30B	93.5 (86.7)	95.5 (88.9)	81.1 (60.9)
Gemma3-27B	79.3 (74.8)	88.9 (81.8)	63.4 (46.6)
Llama4-Scout-17B	92.5 (82.3)	91.9 (85.9)	79.8 (64.3)

Table 2: Hard-slice accuracy (%) across instruction-tuned backbones. Each cell reports JCA accuracy with the strongest primary baseline in parentheses, where primary baselines are Direct, CoT, Compare, and NLI. JCA achieves the highest hard-slice accuracy in all nine dataset–backbone combinations. Full comparison grids are reported in Appendix B .

Figure 2: Accuracy versus mention gap on AmbigQA with Llama4-Scout-17B. More negative values indicate that retrieval mentions a plausible distractor more often relative to the gold answer. Baseline accuracy declines as the gap becomes more negative, while JCA remains comparatively stable. Shaded bands show 95% confidence intervals for JCA.

A. Full-set accuracy (%)
	AmbigQA	Extracted	SituatedQA
Constraint Tally	85.3	86.7	74.9
Verdict-only aggregation	89.1	90.5	84.1
Full JCA	93.3	93.6	85.5
B. Step 3 gain by verdict outcome (pp)
Decisive	$-$ 0.5	$-$ 0.7	$-$ 3.2

Table 3: Contribution of JCA’s final aggregation stage with Llama4-Scout-17B. Panel A reports full-set accuracy for three aggregation rules applied to the same Step 2 reports. Panel B shows the accuracy change from Constraint Tally to full JCA within different Step 2 verdict outcomes, based on each candidate’s Constraint Tally score. In Decisive cases the gold candidate scores strictly higher than every distractor. In Tied cases it shares the top score with at least one distractor. In Gold outranked cases a distractor outranks it and at least one candidate has a positive score. Rare no-pass cases, where a distractor outranks the gold candidate and no candidate has a positive score, are omitted from Panel B. Differences in the text are computed from unrounded accuracies.

	Instruction			Reasoning
Dataset	OnePass JCA	CoT	Staged JCA	CoT	Staged JCA
AmbigQA	83.3	86.4	93.5	94.6	95.6
AmbigQA-Extracted	85.9	86.9	95.5	90.4	91.4
SituatedQA	63.4	60.5	81.1	87.4	90.3

Table 4: Hard-slice accuracy (%) with Qwen3-30B under the tested instruction and native-reasoning settings. OnePass collapses JCA’s three operations into a single completion, while staged JCA keeps them explicit. Paired differences and 95% confidence intervals are reported in Appendix E .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	AmbigQA	Extracted	SituatedQA
Before filtering	875	715	792
Removed	79	167	118
Final evaluation set	796	548	674

Appendix

Table 5: Co-correct filtering counts for the final GPT-5.5 judgment pass.

AmbigQA ( $n=796$ )
Method	Qwen3-30B	Gemma3-27B	Llama4-Scout-17B
Direct	88.3	66.0	84.2
CoT	90.7	79.6	87.4
Compare	91.7	74.4	82.9
NLI	90.8	83.9	78.9
JCA	94.0	87.4	93.3

Appendix

Table 6: Full-set accuracy (%) across all three backbones. Evaluation-set sizes are 796 for AmbigQA, 548 for AmbigQA-Extracted, and 674 for SituatedQA. Best accuracy in each column is bolded.

AmbigQA ( $n=294$ )
Method	Qwen3-30B	Gemma3-27B	Llama4-Scout-17B
Direct	82.3	43.5	75.9
CoT	86.4	67.7	82.3
Compare	86.7	56.8	71.1
NLI	85.7	74.8	66.7
JCA	93.5	79.3	92.5

Appendix

Table 7: Hard-slice accuracy (%) across all three backbones. Hard-slice sizes are 294 for AmbigQA, 198 for AmbigQA-Extracted, and 238 for SituatedQA. Slice membership is defined from the fixed top-20 retrieval and is identical across backbones. Best accuracy in each column is bolded.

	AmbigQA			AmbigQA-Extracted			SituatedQA
Method	Gold-fav.	Near-miss-fav.	$\times$	Gold-fav.	Near-miss-fav.	$\times$	Gold-fav.	Near-miss-fav.	$\times$
CoT	9.9%	26.7%	2.7	10.4%	33.3%	3.2	18.1%	32.5%	1.8
CoT-thinking	3.6%	17.3%	4.8	2.1%	20.1%	9.7	9.2%	20.3%	2.2
OnePass JCA	6.3%	20.0%	3.2	5.6%	35.2%	6.3	16.1%	33.3%	2.1
Staged JCA	8.1%	16.0%	2.0	9.0%	19.5%	2.2	17.3%	23.8%	1.4
Merged-check JCA	4.5%	16.0%	3.6	5.6%	26.6%	4.7	12.5%	24.7%	2.0

Appendix

Table 8: Near-miss selection at the endpoints of the controlled retrieval-support intervention. “Gold-fav.” and “Near-miss-fav.” denote the gold-favored and near-miss-favored endpoints, respectively. AmbigQA and AmbigQA-Extracted compare

4{:}1

with

1{:}4

support, and SituatedQA compares

2{:}1

with

1{:}2

. The final column within each dataset gives the multiplicative increase in near-miss selection.

Candidate	Verdict pattern	Abridged report content
Gilda Radner (gold)	All Satisfies	Gilda Radner played Emily Litella, the character who repeatedly said “Never mind” after being corrected.
Emily Litella	All Satisfies	Emily Litella is the character on Saturday Night Live who says “Never mind.”

Appendix

Table 9: Illustrative tied-verdict case. The Step 2 verdict patterns are identical, while the report content preserves the distinction between the actress and the character she played. Report content is abridged for readability. Full JCA selects the gold answer, Gilda Radner.

Dataset	Backbone	Full $\Delta$ [95% CI]	Hard $\Delta$ [95% CI]
AmbigQA	Qwen3-30B	$-4.02$ [ $-6.03$ , $-2.01$ ]	$-10.20$ [ $-14.29$ , $-6.46$ ]
	Llama4-Scout-17B	$-8.29$ [ $-10.93$ , $-5.78$ ]	$-16.67$ [ $-21.77$ , $-11.56$ ]
AmbigQA-Extracted	Qwen3-30B	$-4.38$ [ $-6.75$ , $-2.19$ ]	$-9.60$ [ $-14.14$ , $-5.05$ ]
	Llama4-Scout-17B	$-5.84$ [ $-8.58$ , $-3.28$ ]	$-10.10$ [ $-15.66$ , $-5.05$ ]
SituatedQA	Qwen3-30B	$-8.31$ [ $-11.87$ , $-4.90$ ]	$-17.65$ [ $-23.53$ , $-11.76$ ]
	Llama4-Scout-17B	$-10.39$ [ $-13.80$ , $-7.12$ ]	$-14.29$ [ $-20.17$ , $-8.40$ ]

Appendix

Table 10: OnePass JCA relative to staged JCA on the full evaluation sets and hard slices. Values are accuracy differences in percentage points, OnePass

-

staged JCA, with 95% paired-bootstrap confidence intervals. All twelve comparisons significantly favor staged JCA.

Dataset	Backbone	$n$	$\Delta$ [95% CI]	Result
AmbigQA	Qwen3-30B	796	$-0.13$ [ $-1.88$ , $+1.63$ ]	NI
	Gemma3-27B	796	$-0.13$ [ $-2.76$ , $+2.51$ ]	Inconclusive
	Llama4-Scout-17B	796	$-0.13$ [ $-2.01$ , $+1.76$ ]	NI
AmbigQA-Extracted	Qwen3-30B	548	$-1.64$ [ $-3.28$ , $+0.00$ ]	Inconclusive
	Gemma3-27B	548	$-0.36$ [ $-3.47$ , $+2.74$ ]	Inconclusive
	Llama4-Scout-17B	548	$-1.28$ [ $-3.28$ , $+0.73$ ]	Inconclusive

Appendix

Table 11: Merged-check JCA versus staged JCA on the full evaluation sets.

\Delta

is merged

-

staged accuracy in percentage points. NI is certified when the lower bound of the 95% confidence interval exceeds

-2.5

points, and “Superior” denotes an interval entirely above zero. Per-dataset rows are secondary comparisons, while the pooled within-backbone rows provide the primary non-inferiority tests.

Dataset	Reasoning off $\Delta$ [95% CI]	Reasoning on $\Delta$ [95% CI]	Reasoning-on NI
AmbigQA	$\mathbf{+7.14}$ [ $+3.74$ , $+10.88$ ]	$+1.02$ [ $-2.04$ , $+4.08$ ]	Yes
AmbigQA-Extracted	$\mathbf{+8.59}$ [ $+4.55$ , $+13.13$ ]	$+1.01$ [ $-4.55$ , $+6.57$ ]	Not certified
SituatedQA	$\mathbf{+20.59}$ [ $+14.29$ , $+26.89$ ]	$+2.94$ [ $-0.84$ , $+6.72$ ]	Yes

Appendix

Table 12: Staged JCA relative to CoT under the two tested Qwen3-30B reasoning regimes on the hard slice. Differences are staged JCA

-

CoT in percentage points. NI refers to staged JCA relative to CoT, both with the Thinking checkpoint, at a 2.5-point non-inferiority margin.

Dataset	Method	Req./inst.	Input tok.	Output tok.	Seq. lat. (s)	Cand.-par. lat. (s)
AmbigQA	NLI	2.24	94,069	4	9.16	8.91
	Staged JCA	4.24	96,030	733	28.77	13.85
	Merged-check JCA	3.00	43,090	888	19.32	19.32
	CoT-thinking	1.00	41,485	1,424	23.85	23.85
SituatedQA	NLI	2.00	82,274	4	9.14	8.94
	Staged JCA	4.00	84,198	872	27.71	15.27

Appendix

Table 13: Average per-instance inference cost on 25 AmbigQA and 25 SituatedQA instances using Qwen3-30B-A3B with greedy decoding (Instruct checkpoint for JCA and NLI, Thinking checkpoint for CoT-thinking). Measurements were collected with one request in flight. Candidate-parallel latency is derived from the measured call latencies as described in Appendix E.4 and is not a wall-clock measurement under concurrent serving. Step 1 constraint-extraction tokens were not instrumented, although its request is included in the request count.

Reranker	Dataset	Max Full	Max Hard	Mean Full	Mean Hard
MiniLM	AmbigQA	59.3	33.0	54.6	19.4
	AmbigQA-Extracted	57.7	41.9	52.4	24.2
	SituatedQA	55.0	27.3	52.5	25.2
BGE-large	AmbigQA	59.5	41.5	50.6	20.4
	AmbigQA-Extracted	56.4	52.3	49.8	27.3
	SituatedQA	55.8	44.5	54.3	36.6

Appendix

Table 14: Cross-encoder reranker accuracy (%) on the final evaluation sets. “Max” and “Mean” aggregate passage-level scores for each candidate by the corresponding operation. Full/hard set sizes are 796/294 for AmbigQA, 548/198 for AmbigQA-Extracted, and 674/238 for SituatedQA.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Controllable Evidence Selection in Retrieval-Augmented Question Answering via Deterministic Utility Gating
Feb 23, 2026Victor P. UndaEvidence Selection Retrieval-Augmented Generation
Independent Researcher, Applied AI Systems
Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Aug 24, 2026Jingjie Ning, Xueqi LiRAG Evaluation Retrieval-Augmented Generation
School of Computer Science, Carnegie Mellon University Pittsburgh, Pennsylvania, USA
Regime Boundary Alignment for Evidence-Gated Question Answering
Sep 25, 2026Zeyan Li, Qirong Guo, SIyuan Qiu +3Evidence-Grounded QA LLM Grounding
Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou)

Feb 23, 2026J/K move · Enter open · S save

Controllable Evidence Selection in Retrieval-Augmented Question Answering via Deterministic Utility Gating

Victor P. Unda

Independent Researcher, Applied AI Systems

Evidence Selection Retrieval-Augmented Generation Knowledge-Intensive QA

Open paper AskPDF arXiv

Many modern AI question-answering systems convert text into vectors and retrieve the closest matches to a user question. While effective for topical similarity, similarity scores alone do not explain why some retrieved text can serve as evidence while other equally similar text cannot. When many candidates receive similar scores, systems may select sentences that are redundant, incomplete, or address different conditions than the question requires. This paper presents a deterministic evidence selection framework for retrieval-augmented question answering. The approach introduces Meaning-Utility Estimation (MUE) and Diversity-Utility Estimation (DUE), fixed scoring and redundancy-control procedures that determine evidence admissibility prior to answer generation. Each sentence or record is evaluated independently using explicit signals for semantic relatedness, term coverage, conceptual distinctiveness, and redundancy. No training or fine-tuning is required. In the prototype, a unit is accepted only if it explicitly states the fact, rule, or condition required by the task. Units are not merged or expanded. If no unit independently satisfies the requirement, the system returns no answer. This deterministic gating produces compact, auditable evidence sets and establishes a clear boundary between relevant text and usable evidence.

Abstract

Retrieval-Augmented Generation

Explore similar work

Controllable Evidence Selection in Retrieval-Augmented Question Answering via Deterministic Utility Gating

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

Regime Boundary Alignment for Evidence-Gated Question Answering