EXCISE: Locating and Excising Distracting Sentences in Medical Question Answering

Yerim Hong; Sohyun Son; Heeseung Chung; Mujeen Sung

cs.CLOct 26, 2026

EXCISE: Locating and Excising Distracting Sentences in Medical Question Answering

Authors: Yerim Hong, Sohyun Son, Heeseung Chung, Mujeen Sung

Organizations: Kyung Hee University

Abstract

Language models are increasingly considered for clinical decision support, where their answers should be grounded in clinical evidence and remain robust to irrelevant information. Prior work shows that a single irrelevant sentence can lower the accuracy of medical models, but not where inside the model its effect can be removed, or how to do so without knowing which sentence it is. We find that blocking attention to a known distractor restores the answer, whereas head masking can suppress correct answers. We also show that the model's hidden states mark the distractor as the sentence that least fits the case even when the model follows it. To remove a distractor's effect without knowing which sentence it is, we propose EXCISE, an inference-time method that blocks the sentence most influencing the answer, but only if the model's hidden states mark it as inconsistent with the case. On MedQA-TD (MedQA with Targeted Distractors), a benchmark we construct in which each distractor targets a specific wrong option, EXCISE gains 16.7-22.8 accuracy points at a 1.5-2.1-point cost on the same questions without distractors, outperforming prompting, rewriting, and steering baselines for all three models. Its gains extend to MedDistractQA and BiasMedQA, as well as chain-of-thought generation.

Figures & tables

Figure 1: Excise on an example from MedQA-TD with Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 ) . The distractor s6 shifts the answer to B; Excise selects s6 and blocks attention to it, restoring C.

General
Model	MedQA	MDQA	MedQA-TD
Llama-3.1-8B-Instruct	61.7	54.5 $-$ 7.2	40.7 $-$ 21.1
Qwen3-8B	63.4	50.3 $-$ 13.1	33.7 $-$ 29.7
Llama-3-8B-Instruct	60.0	49.7 $-$ 10.3	36.2 $-$ 23.8
Granite-3.3-8B-Instruct	51.5	39.4 $-$ 12.1	29.1 $-$ 22.4
Falcon3-10B-Instruct	58.4	46.3 $-$ 12.1	31.5 $-$ 27.0

Table 1: Accuracy (%) on MedQA ( Jin et al., 2021 ) without and with distractors, averaged over the two distractor types (MDQA: MedDistractQA ( Vishwanath et al., 2025a ) ), with the change from MedQA in gray. † Reasoning model.

Figure 2: Head masking and blocking the added sentence (in every layer and head) when the sentence points to a wrong option ( MedQA-TD ) or the correct option, averaged over distractor types and the four models.

Figure 3: How often a sentence is the most discordant in its case: the sentence that moves the answer most in MedQA questions (left), and the distractor in questions it turns from correct to wrong (right). Dashed lines: chance.

Figure 4: An example from MedQA-TD with Llama-3.1-8B-Instruct (distractor s6 in red, targeting B, gold: C). Top right: attribution $a_{i}$ and discordance $d_{i}$ of each sentence (dashed line: threshold). Bottom: without intervention, the chain of thought ( Wei et al., 2022 ) calls the parrot a red herring yet uses it as a clue and answers B. With s6 blocked in all layers and heads, the model answers C (excerpts).

Qwen3-8B
Method	Latency	Clean		With distractors
Method	(ms)	MedQA	MMLU-PM	MedQA-TD	MedDistractQA	BiasMedQA
No intervention	41	63.4	82.7	33.7	50.3	28.7
Ignore instruction	44	64.0 +0.6	82.7 0.0	33.2 $-$ 0.5	50.4 +0.1	27.1 $-$ 1.6
CAA	41	64.0 +0.6	80.5 $-$ 2.2	38.3 +4.7	53.3 +3.0	35.6 +6.9
S2A	4,182	63.6 +0.2	83.5 +0.7	36.5 +2.8	54.3 +4.0	35.3 +6.7

Table 2: Accuracy (%) of Excise and inference-time baselines. MedQA-TD and MedDistractQA: mean of the Nonliteral and Bystander distractors (per type in Appendix F.1 ); BiasMedQA: mean of seven bias types. Gray: change from no intervention, or from CoT below the dashed line (unrounded), shaded blue for gains and red for losses. Latency: wall-clock time per question, including generation. MMLU-PM: MMLU professional medicine. ‡ Model-card prompt format (Appendix F.3 ). Bold : best per model and column within each block.

Variant	#Fwd	MedQA		MedQA-TD
Variant	#Fwd	Acc	Interv.	Acc	Interv.
Qwen3-8B
Excise (Ours)	25	61.7	19.4	56.4	48.6
w/o IA	18	61.7 $-$ 0.1	92.5	56.0 $-$ 0.4	98.3
w/o DG	9	53.2 $-$ 8.6	93.5	47.2 $-$ 9.3	98.6
w/o IA & DG	1	63.4 +1.6	–	33.7 $-$ 22.8	–

Table 3: Our ablation study. Interv.: questions intervened on (%); #Fwd: forward passes per question. w/o IA & DG: No intervention

Figure 5: Gains of S2A and Excise on MedQA-TD (both distractor types), by whether the S2A rewrite keeps the distractor.

Figure 6: Accuracy on MedQA-TD (NL and BS pooled) with zero to three added off-case sentences.

Figure 7: Flipped questions recovered by Excise w/o IA and Excise , pooled over MedQA-TD , MedDistractQA and BiasMedQA. Only questions in which the distractor passes the gate are included, grouped by whether the distractor or another sentence ranks first by discordance.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Constructing MedQA-TD for one MedQA question (gold: C). Each bar is one of the eight distractors written for a wrong option, with the reference model’s preference for its strongest wrong option over the gold option. The flipping distractor with the largest margin is kept (red).

General
	Accuracy (%)					Target-wrong answers (%)
		MDQA		MedQA-TD		Nonliteral		Bystander
Model	Clean	NL	BS	NL	BS	Clean	MedQA-TD	Clean	MedQA-TD
Llama-3.1-8B-Instruct	61.7	53.1	55.9	39.2	42.1	17.4	45.8	17.8	42.6
Qwen3-8B	63.4	49.4	51.1	36.4	31.0	19.0	53.0	17.7	59.5
Llama-3-8B-Instruct	60.0	49.6	49.9	37.0	35.4	18.9	49.1	17.0	48.2

Appendix

Table 4: The nine open models without intervention on the 1,273 test questions: Llama-3.1-8B-Instruct and Llama-3-8B-Instruct ( Grattafiori et al., 2024 ) , Qwen3-8B ( Yang et al., 2025a ) , Granite-3.3-8B-Instruct ( Granite Team, 2025 ) , Falcon3-10B-Instruct ( Team, 2024 ) , Phi-4 ( Abdin et al., 2024 ) , MedGemma-27B-text-it ( Sellergren et al., 2025 ) and HuatuoGPT-o1-8B ( Chen et al., 2025 ) . Accuracy: clean and with the MedDistractQA (MDQA) and MedQA-TD distractors of both types (NL, BS). Target-wrong answers: answers selecting the option the distractor was built to promote, on the clean questions and on MedQA-TD . † Reasons before answering and is read from its generated answer (unparsable outputs, at most 0.8%, count as wrong).

Figure 9: What makes MedQA-TD hard (Llama-3.1-8B-Instruct and Qwen3-8B). (a) Target-wrong rate among questions answered correctly without the distractor, by quartile of the clean margin between the gold and the targeted option. (b) Accuracy on all 1,273 questions with a random distractor (random target and phrasing), a random phrasing for the selected target, and the selected distractor ( MedQA-TD ). (c) Target-wrong rate on the questions of (a), by third of the case-free association: the increase in the target-minus-gold logit when the distractor is added to the question sentence and options alone.

Position	Data	Model	Excise	Last	Known distractor
End	MedQA-TD NL	Llama-3.1-8B-Instruct	+19.0	+21.6	+22.1
	MedQA-TD NL	Qwen3-8B	+22.3	+25.3	+25.9
	MedQA-TD BS	Llama-3.1-8B-Instruct	+14.5	+18.6	+19.1
	MedQA-TD BS	Qwen3-8B	+23.3	+30.9	+31.7
	MedQA-TD NL	Llama-3-Meerkat-8B	+23.3	+25.3	+25.8
	MedQA-TD BS	Llama-3-Meerkat-8B	+19.6	+24.5	+25.4

Appendix

Table 5: Accuracy change (points) by the position of the distractor, on held-out data. End: MedQA-TD as built, where the distractor is the last narrative sentence in 97% of questions; Middle: the same test questions with the distractor moved to the middle of the narrative, and BiasMedQA ( Schmidgall et al., 2024 ) confirmation, whose cue is the second narrative sentence; First: the same test questions with the distractor as the first sentence. Last: block the last narrative sentence in every question; Known distractor: block the distractor itself; both with the same eligibility, blocking and full denominators as Excise . The End row of Known distractor is therefore slightly below the gains of Section 4 , which block in every question.

Qwen3-8B
Method	MedQA-TD NL				MedQA-TD BS
	End	Middle	First	Mean	End	Middle	First	Mean
No intervention	36.4	44.7	46.9	42.7	30.9	39.5	41.5	37.3
Ignore instruction	35.4	44.9	46.8	42.4	31.0	39.7	41.6	37.4
CAA	40.5	47.2	49.6	45.7	36.2	44.1	43.0	41.1
Avg. Attention (top-1)	33.9	42.1	47.8	41.3	26.7	34.2	41.2	34.0

Appendix

Table 6: Accuracy (%) with the distractor at the three positions of Table 5 (End: MedQA-TD as built, as in the main tables; Middle; First) and their mean, for the first-token methods of Table 2 , the top-1 attribution baselines of Table 20 and the variants of Table 3 . S2A and the chain-of-thought rows are not run at the other positions. The same 1,273 test questions and the same sentence are used in each position. AT2 is Llama-3.1-8B-Instruct only (–).

Model	MedQA-TD NL				MedQA-TD BS
	End	Middle	First	Mean	End	Middle	First	Mean
Llama-3.1-8B-Instruct	39.2	47.0	47.4	44.5	42.1	49.6	49.2	47.0
Qwen3-8B	36.4	44.7	46.9	42.7	30.9	39.5	41.5	37.3
Llama-3-8B-Instruct	37.0	44.0	44.4	41.8	35.4	45.1	44.9	41.8
Granite-3.3-8B-Instruct	30.6	37.4	36.3	34.8	27.5	34.5	31.6	31.2
Falcon3-10B-Instruct	33.9	40.8	40.7	38.4	29.1	36.8	35.7	33.9

Appendix

Table 7: Accuracy (%) of the nine open models without intervention, with the distractor at the three positions (End: MedQA-TD as built). † Read from its generated answer, as in Table 4 .

Set	Source	Questions	Options	Sentences	Words	$\geq$ 3 (%)
Development
Clean	MedQA val. ( Jin et al., 2021 )	1,269	4	7.0	97.7	94.1
MedQA-TD Nonliteral	as above	1,266	4	7.9	117.6	98.7
MedQA-TD Bystander	as above	1,267	4	8.0	114.1	98.6
Test
Clean	MedQA test ( Jin et al., 2021 )	1,273	4	7.1	100.2	93.5

Appendix

Table 8: Development and test sets. Sentences: mean number of narrative sentences (the case before the question sentence); Words: mean number of words in them;

\geq

3: share of questions with at least three narrative sentences, the only ones on which Excise can intervene.

				Clean		MedQA-TD
						NL		BS
Model	Layer	$\tau$	$\gamma$	None	Excise	None	Excise	None	Excise
Llama-3.1-8B-Instruct	9	1	0	68.40	67.06	42.89	65.24	45.46	63.38
Qwen3-8B	25	1	0	63.51	61.62	34.44	60.03	28.41	54.70
Llama-3-Meerkat-8B	19	0.5	0	67.85	65.17	37.20	64.14	37.88	62.19
Phi-4	16	1	0.01	70.06	68.40	51.34	67.22	49.88	64.80

Appendix

Table 9: Hyperparameters of Excise chosen on the development set for each model, and development accuracy (%) without intervention (None) and with Excise .

\tau

: gate threshold;

\gamma

: blocking strength.

Qwen3-8B
Excise minus	Clean	MedQA-TD NL	MedQA-TD BS
No intervention	$-$ 1.6 [ $-$ 3.0, $-$ 0.3]	+22.3 [+19.9, +24.7]	+23.3 [+20.7, +25.8]
Avg. Attention + DG	$-$ 1.1 [ $-$ 2.4, +0.2]	+21.1 [+18.7, +23.4]	+23.2 [+20.7, +25.7]
CAMAB + DG	$-$ 0.5 [ $-$ 1.5, +0.5]	+3.9 [+2.6, +5.3]	+2.7 [+1.5, +4.0]
ContextCite + DG	$-$ 0.5 [ $-$ 1.4, +0.3]	+0.5 [ $-$ 0.3, +1.3]	$-$ 0.6 [ $-$ 1.6, +0.2]
Excise w/o DG	+8.6 [+6.0, +11.2]	+11.2 [+9.0, +13.6]	+7.3 [+4.9, +9.8]

Appendix

Table 10: Paired comparisons on the test sets: accuracy of Excise minus that of each method (points), with 95% bootstrap intervals over the 1,273 questions (10,000 resamples). Dashed rules separate no intervention, attribution methods with the Discordance Gate (DG), and single modules; AT2 is Llama-3.1-8B-Instruct only.

Qwen3-8B
		Clean	MedQA-TD		Gold-cue control
Method	Heads	MedQA	NL	BS	NL	BS
No mask	–	64.65	37.39	32.68	81.07	85.07
Head mask, tuned on development data	126	54.83	55.15	55.38	38.18	34.17
Random heads, same layers (mean of 5)	126	50.95	26.35	22.70	76.81	80.97
No intervention	–	63.39	36.37	30.95	82.17	86.33

Appendix

Table 11: Head masks from the head search of Liu and Chu (2026) and Excise (accuracy, %). Tuned: chosen on development questions as described above; gold-cue control: the distractor points at the correct option. Random heads: as many heads removed from each layer as in the tuned mask, chosen at random (mean of five masks). Masked and unmasked rows are read in the same run. Mask rows use the forced-choice readout of the head search and Excise rows the first-token readout of Table 2 , so each is compared with the unmodified row of its own block.

Qwen3-8B
		MedQA-TD
Batch for $P$	Clean	NL	BS	BMQ
No intervention	63.39	36.37	30.95	28.67
Test set ( Excise )	61.74	58.68	54.20	53.50
Batches of 50 questions	61.78	58.76	53.74	53.31
Fixed: dev. MedQA + MedQA-TD	62.37	56.01	49.73	47.86

Appendix

Table 12: Accuracy (%) of Excise with the projection of the Discordance Gate estimated on different unlabeled batches. Batches of 50 questions: mean over 20 random partitions of each set. Fixed: estimated once on development (dev.) MedQA and MedQA-TD questions without labels and applied unchanged to every test set. BMQ: BiasMedQA, mean of seven types.

Replacing the representation of DG while IA stays fixed
Variant	#Fwd $\downarrow$	Clean		MedQA-TD NL		MedQA-TD BS
Variant	#Fwd $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$
Qwen3-8B
Hidden states (Ours)	25	61.74	19.40	58.68	47.84	54.20	49.33
SimCSE	9 ‡	61.98	18.46	53.34	41.87	49.96	40.85
MPNet †	9 ‡	62.14	15.16	50.67	35.43	41.08	24.51

Appendix

Table 13: The Discordance Gate computed from an external sentence encoder’s embeddings instead of the model’s hidden states, with Interventional Attribution and the blocking unchanged (columns as in Table 3 ): SimCSE ( Gao et al., 2021 ) with its threshold chosen on the development set, MPNet (all-mpnet-base-v2, a Sentence-BERT model; Song et al., 2020 ; Reimers and Gurevych, 2019 ) and SapBERT ( Liu et al., 2021 ) ; † threshold not tuned (that of Excise ); ‡ model forward passes for IA alone, plus two passes of the encoder per sentence.

(a) How the gate sees the distractor (% of flipped questions).
Set	Attacked	Too short	(i) Flagged first	(ii) Flagged, not first	(iii) Not flagged
Qwen3-8B
MedQA-TD NL	353	2.5	87.3	4.8	5.4
MedQA-TD BS	421	1.9	69.1	11.9	17.1
MedDistractQA NL	209	2.4	90.9	4.3	2.4
MedDistractQA BS	202	2.0	83.7	5.4	8.9

Appendix

Table 14: How the gate sees the distractor on the flipped questions, answered correctly without the distractor and wrongly with it, with the distractor located. Too short: fewer than three narrative sentences, so the gate is not applied. (i) Flagged first: the distractor passes the gate and is the most discordant sentence. (ii) Flagged, not first. (iii) Not flagged.

Model		+	None	w/o IA	Excise	Known	Excise $-$ w/o IA
Llama-3.1-8B-Instruct	NL	1	42.7	45.2	53.3	60.2	+8.1 [6.1, 10.2]
	NL	2	44.2	46.9	52.7	60.8	+5.9 [4.0, 7.8]
	NL	3	45.0	46.3	53.3	59.9	+7.0 [5.1, 9.0]
	BS	1	45.4	45.7	51.9	60.1	+6.2 [4.4, 8.1]
	BS	2	45.3	46.3	51.8	58.5	+5.6 [3.8, 7.3]
	BS	3	45.5	47.0	51.0	59.0	+4.0 [2.4, 5.7]

Appendix

Table 15: Accuracy (%) on MedQA-TD with added off-case sentences. +: number of added sentences. None: no intervention. w/o IA: Excise w/o IA. Known: the question’s own distractor blocked. Last column: difference in points with paired 95% bootstrap interval.

Excise intervenes on (%, questions in the group)
	Llama-3.1-8B-Instruct		Qwen3-8B		Llama-3-Meerkat-8B
Answer without intervention	NL	BS	NL	BS	NL	BS
Target, pulled by the distractor	88.5 (390)	81.0 (343)	88.5 (443)	79.0 (538)	94.0 (450)	88.6 (437)
Target, already on the clean question	67.4 (193)	53.3 (199)	63.4 (232)	57.5 (219)	71.5 (200)	68.3 (189)
Wrong, not the target	27.7 (191)	33.3 (195)	25.2 (135)	26.2 (122)	31.5 (146)	38.0 (163)
Correct	9.6 (499)	15.3 (536)	7.8 (463)	11.4 (394)	20.1 (477)	21.7 (484)

Appendix

Table 16: How often Excise intervenes (%; questions in parentheses), by the answer each model gives on MedQA-TD without intervention. Pulled by the distractor: the target on MedQA-TD but not on the clean question; already on the clean question: the target even without the distractor. Groups use the target labels, which Excise never sees. Lower block: target-wrong answers among the questions on which Excise intervenes or not.

Llama-3.1-8B-Instruct
	IA ranks	DG	Blocking	Recovered
Set	it first	passes it	restores	Excise	Known
MedQA-TD NL	90	95	95	81	94
MedQA-TD BS	90	86	93	73	92
MedDistractQA NL	72	93	90	62	88
MedDistractQA BS	76	97	90	67	85

Appendix

Table 17: Steps of Excise on flipped questions from MedQA-TD and MedDistractQA (%). The first three columns give successive pass rates. Recovered: the share of all flipped questions answered correctly after Excise or blocking the known distractor.

Qwen3-8B
Method	#Fwd	s/q	Clean		MedQA-TD		MedDistractQA		BiasMedQA
Method	#Fwd	s/q	MedQA	MMLU-PM	NL	BS	NL	BS	Avg.
No intervention	1	0.041	63.39	82.72	36.37	30.95	49.41	51.14	28.67
Ignore instruction	1	0.044	64.02	82.72	35.35	31.03	49.49	51.22	27.08
CAA	1	0.041	64.02	80.51	40.46	36.21	51.77	54.83	35.57
S2A †	–	4.182	63.63	83.46	39.75	33.23	55.93	52.63	35.34

Appendix

Table 18: Accuracy (%) by distractor type for the methods in Table 2 . MMLU-PM: MMLU professional medicine. BiasMedQA: mean of seven bias types. NL / BS: Nonliteral / Bystander. #Fwd: forward passes per question, not counting the generation of S2A † ( Weston and Sukhbaatar, 2023 ) . s/q: seconds per question, including generation. Below the dashed line: chain-of-thought generation (Appendix F.3 ), with I 3 C ( Wu et al., 2024 ) . ‡ Prompted in the answer format of its model card (Appendix G ). Bold : best per model and column, above and below the dashed line separately.

Qwen3-8B
Variant	#Fwd $\downarrow$	Clean		MedQA-TD NL		MedQA-TD BS
Variant	#Fwd $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$	Acc $\uparrow$	Interv. rate $\downarrow$
IA (Ours)	25	61.74	19.40	58.68	47.84	54.20	49.33
Avg. Attention	18	62.84	35.74	37.63	14.53	31.03	20.03
AT2	–	–	–	–	–	–	–
CAMAB	58	62.22	19.95	54.75	39.59	51.45	43.05

Appendix

Table 19: Replacing IA while keeping the Discordance Gate fixed. Acc: accuracy (%); Interv.: questions intervened on (%); #Fwd as in Table 2 . AT2 is evaluated on Llama-3.1-8B-Instruct only (–).

Qwen3-8B
Method	#Fwd	s/q	Clean		MedQA-TD		MedDistractQA		BiasMedQA
Method	#Fwd	s/q	MedQA	MMLU-PM	NL	BS	NL	BS	Avg.
Avg. Attention	2	0.067	60.25	81.99	33.86	26.71	46.11	46.98	45.89
CAMAB	42	1.39	55.93	73.90	46.35	47.29	49.73	49.49	43.33
ContextCite	66	1.84	53.57	71.69	47.45	48.23	48.63	47.53	43.77
Llama-3.1-8B-Instruct

Appendix

Table 20: Context attribution methods that block their top-ranked sentence in every question (accuracy, %; columns as in Table 2 , where Excise and the other baselines are reported).

Figure 10: Gains of S2A ( Weston and Sukhbaatar, 2023 ) by whether the model’s rewrite keeps the distractor ( MedQA-TD ). (a) Share of S2A rewrites that keep the distractor (at least 80% of its content words) against the gain of S2A over no intervention, for eight instruct models and both distractor types. (b, c) Gains of S2A and Excise on the questions whose rewrite keeps (b) or drops (c) the distractor, for the four models run with both methods (Nonliteral and Bystander pooled;

n

: questions).

	MedQA-TD NL					MedQA-TD BS					Clean MedQA
Model	Kept	None	S2A	Excise	S2A+ Excise	Kept	None	S2A	Excise	S2A+ Excise	None	S2A	Excise	S2A+ Excise
Llama-3.1-8B-Instruct	16.1	39.2	58.3	58.2	57.8	37.4	42.1	53.1	56.6	54.9	61.7	61.1	59.6	59.3
Qwen3-8B	85.9	36.4	39.7	58.7	59.3	90.8	31.0	33.2	54.2	54.8	63.4	63.6	61.7	62.1
Phi-4	8.4	51.7	67.5	66.8	66.5	12.4	49.6	66.1	65.2	65.5	70.9	71.5	69.1	68.7
OLMo-2-7B-Instruct	98.9	23.3	24.3	30.8	26.6	99.6	16.4	17.7	25.7	20.4	42.4	–	41.2	–

Appendix

Table 21: S2A ( Weston and Sukhbaatar, 2023 ) , Excise , and Excise applied to the model’s own S2A rewrite (S2A+ Excise ; an exploratory combination). Accuracy (%). Kept: share (%) of S2A rewrites that keep the distractor, i.e., contain at least 80% of its content words. Bold : highest accuracy per model and set; –: S2A rewrites not produced for OLMo-2-7B-Instruct’s clean questions.

	Llama-3.1-8B-Instruct					Qwen3-8B
Bias type	No int.	Excise	$\Delta$ [95% CI]	IA top-1	DG flag	No int.	Excise	$\Delta$ [95% CI]	IA top-1	DG flag
Recency	49.6	54.3	+4.7 [+3.1, +6.4]	26	95	25.9	52.9	+27.0 [+24.5, +29.6]	63	93
Confirmation	59.1	57.7	$-$ 1.3 [ $-$ 2.7, +0.1]	19	96	41.1	55.9	+14.8 [+12.5, +17.1]	39	94
Frequency	38.2	52.4	+14.2 [+12.0, +16.4]	42	95	25.6	52.9	+27.3 [+24.7, +29.9]	65	89
Status quo	44.4	54.1	+9.7 [+7.7, +11.9]	36	94	27.7	53.1	+25.5 [+22.9, +28.1]	63	93
False consensus	52.0	56.8	+4.8 [+3.0, +6.6]	26	97	20.3	51.4	+31.0 [+28.3, +33.7]	74	90

Appendix

Table 22: BiasMedQA ( Schmidgall et al., 2024 ) by bias type (1,273 five-option questions each): accuracy (%) and difference (points) with 95% paired bootstrap intervals. IA top-1 / DG flag: among questions whose bias cue is a separate narrative sentence (93–99%), the share in which attribution ranks it first and in which the gate flags it. Clean: the same questions without the cue.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 493 papers

An Empirical Study of Development-Tuned Representation Sensitivity for Medical QA Data Selection
Sep 9, 2026Yuexin Wu, Dayou Yu, Vasile RusData Selection Medical QA
Department of Computer Science, University of Memphis
HypothesisMed: Inference-Time Answer Fusion and Structured Hypothesis-Space Reporting for Biomedical Question Answering
May 31, 2026Md Motaleb Hossen Manik, Ge WangBiomedical QA LLM Uncertainty Estimation
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA · Department of Biomedical Engineering Rensselaer Polytechnic Institute Troy, NY, USA
Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution
Mar 5, 2026Qiao Jin, Yin Fang, Lauren He +12LLM Answer Verification Healthcare
Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health +3

Sep 9, 2026J/K move · Enter open · S save

An Empirical Study of Development-Tuned Representation Sensitivity for Medical QA Data Selection

Yuexin Wu, Dayou Yu, Vasile Rus

Department of Computer Science, University of Memphis

Data Selection Medical QA Language Model Robustness

Open paper AskPDF arXiv

We study medical question-answering (QA) data selection when a developer already has questions, labels, and auxiliary explanatory texts but can retain only

B

examples for a final LoRA update. Representation-Sensitivity Prioritization (RSP) first restricts the candidate bank to its top-

4B

predictive-entropy examples, then combines the frequency of answer flips with non-flipping contraction of the top-two answer margin under hidden-state perturbations. We evaluate this operational score on five medical QA datasets with google/medgemma-4b-it,

B=128

, and three seed branches. After removing four MedExpQA test questions that also occur in its acquisition pool, dataset-specific development-tuned RSP attains the highest observed five-dataset mean accuracy (53.44%) and Macro-F1 (48.21%) among the evaluated selectors. Relative to random selection within the same entropy-restricted pool, the descriptive differences are +1.49 and +2.10 percentage points, respectively; paired 95% bootstrap intervals include zero. The audit mixes benchmarks with different relationships to documented backbone post-training sources, and the same test cohorts had been used in earlier analyses before the expanded tuning grid was designed. Results are therefore exploratory, modest, and dataset-dependent rather than evidence of statistical superiority. RSP point estimates also exceed an archived no-update reference, but unpinned model revisions and incomplete historical input identifiers limit attribution of that difference to fine-tuning. Because every selected training example includes auxiliary text and no matched answer-only control is evaluated, the experiments also do not identify a causal benefit from auxiliary-text conditioning itself.

Abstract

Language Model Robustness

Explore similar work

An Empirical Study of Development-Tuned Representation Sensitivity for Medical QA Data Selection

HypothesisMed: Inference-Time Answer Fusion and Structured Hypothesis-Space Reporting for Biomedical Question Answering

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution