BIT.UA at BioASQ 14B: Modular Retrieval with pg_textsearch and Qdrant, and Agent-Based Answer Generation

André Ribeiro; Rúben Garrido; Alexander Christiansen; Richard A. A. Jonker; Sérgio Matos

cs.CLSep 4, 2026

BIT.UA at BioASQ 14B: Modular Retrieval with pg_textsearch and Qdrant, and Agent-Based Answer Generation

Authors: André Ribeiro, Rúben Garrido, Alexander Christiansen, Richard A. A. Jonker, Sérgio Matos

Organizations: IT, University of Aveiro, Aveiro, Portugal · IEETA/DETI, LASI, University of Aveiro, Aveiro, Portugal · Aalborg University Business School, Aalborg, Denmark

Abstract

This paper describes the participation of the BIT.UA team from the University of Aveiro in the 14th edition of the BioASQ Task B challenge on biomedical question answering. Building on our previous submissions, we introduced a substantially refactored and modular codebase, and made significant changes to both the retrieval and generation components of the pipeline. For PhaseA document retrieval, we replaced the PyTerrier PISA index with PostgreSQL-based pg_textsearch for BM25 retrieval and adopted Qdrant for dense embedding indexing, enabling more efficient storage and GPU-accelerated similarity search. We explored HyDE-based query expansion alongside a Context-1 retrieval strategy. A new reranker training pipeline was developed, incorporating dense retrieval for negative sampling. For Phases A+ and B answer generation, we introduced an LLM-as-a-judge framework and a novel agent quorum mechanism, where multiple agents with diverse prompts debate and iteratively converge on a consensus answer using adaptive document retention. We also participated in the snippets generation subtask for the first time. Our systems achieved competitive results across all batches, with PhaseA systems achieving MAP ranks of 5 (Batch~1,3). We discuss the impact of these architectural changes, lessons learned, and outline directions for future work including SPLADE and ColBERT integration. All code is openly available: https://github.com/bioinformatics-ua/BioASQ14b.

Figures & tables

Panel of Figure 1 — Figure 1: Pipeline architecture for the 14th edition. The traditional Phase A path (top) fuses BM25 and dense retrieval, re-ranks, and extracts snippets. Context-1 (bottom) is a fully independent agentic pipeline that uses BM25 and dense search as internal tools, producing a ranked document set.

Batch	Version / Config	Agents (Model)
B2	V1 ( 3-open-27-120b )	Qwen3-Next-80B, Nemotron-3-Super-120B, MedGemma-27B
		Synthesizer: Nemotron-3-Super-120B
	V2 ( 5-mixed-api )	Nemotron-3-Super-120B, Grok-4.1-Fast, Gemini-3-Flash, Gemini-2.5-Flash, GPT-5-Mini
		Synthesizer: Claude-Sonnet-4.6
	V3 ( 4-open-35-309b )	Qwen3-Next-80B, Mimo-v2-Flash (309B), Qwen3.5-35B, Nemotron-3-Super-120B
		Synthesizer: GPT-5.4-Nano

Table 1: Agent quorum configurations across test batches. Agents are listed with their model identifier and the synthesizer used for final answer generation. The Config code encodes the number of agents, weight openness ( open = all weights public, mixed = some API-only), and the parameter range (in billions) across open-weight agents; api denotes configurations dominated by hosted closed models.

Method	Docs	M@100	H@100	AR@100	MRR	R@100
BM25 (baseline)	99.3	11.01	0.953	0.082	0.549	0.373
Hybrid RRF	100.0	12.28	0.965	0.047	0.696	0.406
Hybrid RRF + BGE-reranker-base	100.0	12.28	0.965	0.047	0.498	0.406
Hybrid RRF + BGE-reranker-v2-m3	100.0	12.28	0.965	0.047	0.649	0.406
Hybrid weighted sum	100.0	11.47	0.965	0.082	0.622	0.384
Hybrid wsum + BGE-reranker-base	100.0	11.47	0.965	0.082	0.597	0.384

Table 2: Internal retrieval testing results comparing BM25 baseline, fusion strategies of BM25 and dense retrieval (hybrid RRF and weighted sum), reranked variants, and Context-1. Bold values indicate the best result in each column.

Model	Configuration	MAP-bioasq@10
LLaMA Nemotron rerank 1B	E2-S4, multi_neg_pairwise, InfoNCE	0.9995
LLaMA Nemotron rerank 1B	E5, pairwise	0.9970
BGE-reranker-v2-m3	E5, pairwise	0.6824
BGE-reranker-base	E5, pairwise	0.6686
nboost/pt-biobert-base-msmarco	E5, pairwise	0.6608
ncbi/MedCPT-Cross-Encoder	E5, pairwise	0.6404

Table 3: Internal validation results for selected reranker models on the 13B golden datasets, sorted by performance.

Model	Configuration	MAP-bioasq@10
BGE-reranker-v2-m3	basic	0.6489
BGE-reranker-v2-m3	shifter	0.6824
BGE-reranker-base	basic	0.6312
BGE-reranker-base	shifter	0.6686
cross-encoder/ms-marco-MiniLM-L-6-v2	basic	0.5928
cross-encoder/ms-marco-MiniLM-L-6-v2	shifter	0.6373

Table 4: Internal validation results for sampling strategies to reranker training, sorted by performance.

Batch	System	Retriever	Reranker Ensemble	N	MAP	Rank
1	bioinfo-0	BM25	Filtered (MAP $>$ 0.64)	5–7	0.1476	34
	bioinfo-1	BM25 + DPRF	Filtered (MAP $>$ 0.64)	5–7	0.1502	30
	bioinfo-2	BM25 + DPRF	All 2026 rerankers	$\sim$ 24	0.2220	14
	bioinfo-3	BM25 + DPRF	All 2025 rerankers	$\sim$ 46	0.2479	5
	bioinfo-4	BM25 + DPRF	All 2025 + 2026 rerankers	$\sim$ 70	0.2398	11
	Best				0.2835	1

Table 5: Phase A document retrieval results across all four Test Batches. Best is the top competitor MAP (rank 1); Median is the median competitor MAP for that batch.

System	Batch B3		Batch B4
System	MAP	Rank	MAP	Rank
bioinfo-0	0.0546	25	0.0591	28
bioinfo-1	0.0352	32	0.0296	51
bioinfo-2	0.0375	30	0.0495	35
bioinfo-3	0.0267	38	0.0431	42
bioinfo-4	0.0130	47	0.0490	36

Table 6: Snippet retrieval results across test batches. All submissions used the Phase A retrieval pipeline for document sourcing; snippet extraction was performed by the same generation models used in Phase A+/B. System ranks are calculated out of all participating systems (50 in B3, 66 in B4).

Batch	System	Generation Configuration	Y/N F1	Fact. MRR	List F1
1	bioinfo-0	MedGemma-27B + Qwen3.5 35B	0.8211 (37)	0.2696 (23)	0.1509 (28)
	bioinfo-1	Qwen3.5 235B + Qwen Max Thinking	0.8786 (16)	0.3261 (13)	0.1735 (20)
	bioinfo-2	Gemini 2.0 Flash + 2.5 Flash + Qwen3.5 122B	0.8786 (16)	0.3261 (13)	0.1731 (22)
	bioinfo-3	Gemini 2.0 Flash + 2.5 Flash	0.8786 (16)	0.2609 (24)	0.1735 (20)
	bioinfo-4	All models ensemble	0.8786 (16)	0.2609 (24)	0.1731 (22)
	Best		1.0000 (1)	0.4783 (1)	0.2345 (1)

Table 7: Phase A+ exact answer results across all four Test Batches. Scores are shown with their overall leaderboard ranking in parentheses. The top-performing system among our submissions in each batch is highlighted in bold. Best represents the top competitor score (Rank 1); Median is the median score (and median rank) across all participants for that batch.

Batch	System	Generation Configuration	Y/N F1	Fact. MRR	List F1
1	bioinfo-0	LLM-as-judge top-k=3	0.8786 (24)	0.4130 (23)	0.2581 (33)
	bioinfo-1	LLM-as-judge top-k=5	0.8786 (24)	0.4565 (8)	0.2659 (28)
	Best		1.0000 (1)	0.5217 (1)	0.3613 (1)
	Median		0.9377 (21)	0.4203 (21)	0.2443 (21)
2	bioinfo-0	Agents v1 (3-open-27-120b)	0.8518 (35)	0.2500 (39)	0.3836 (28)
	bioinfo-1	Agents v2 (5-mixed)	0.8929 (9)	0.3250 (20)	0.2270 (43)

Table 8: Phase B exact answer results across all four Test Batches. Scores are shown with their overall leaderboard ranking in parentheses. The top-performing system among our submissions in each batch is highlighted in bold. Best represents the top competitor score (Rank 1); Median is the median score (and median rank) across all participants for that batch.

Batch	System	Generation Configuration	R-2 Rec.	R-2 F1	R-SU4 Rec.	R-SU4 F1
1	bioinfo-0	Nemotron-3-Super-120B	0.2150 (21)	0.1021 (35)	0.2340 (18)	0.1094 (33)
	bioinfo-1	Nemotron-3-Super-120B	0.2200 (17)	0.1009 (36)	0.2393 (17)	0.1074 (36)
	bioinfo-2	Opus + Sonnet	0.2408 (7)	0.0825 (48)	0.2706 (6)	0.0932 (48)
	bioinfo-3	Opus + Sonnet	0.2245 (16)	0.0955 (41)	0.2489 (15)	0.1021 (40)
	bioinfo-4	Opus + Sonnet	0.2348 (11)	0.0916 (44)	0.2664 (8)	0.1023 (39)
	Best		0.2725 (1)	0.1767 (1)	0.2949 (1)	0.1821 (1)

Table 9: Phase A+ ideal answer results across all four Test Batches. Scores are shown with their overall leaderboard ranking in parentheses. The top-performing system among our submissions in each batch is highlighted in bold. Best represents the top competitor score (Rank 1); Median is the median score (and median rank) across all participants for that batch.

Batch	System	Generation Configuration	R-2 Rec.	R-2 F1	R-SU4 Rec.	R-SU4 F1
1	bioinfo-0	LLM-as-judge top-k=3	0.2857 (13)	0.1303 (39)	0.3033 (14)	0.1323 (38)
	bioinfo-1	LLM-as-judge top-k=5	0.2678 (20)	0.0958 (55)	0.2805 (21)	0.0982 (55)
	Best		0.3746 (1)	0.2487 (1)	0.3835 (1)	0.2381 (1)
	Median		0.2470 (33)	0.1645 (33)	0.2517 (33)	0.1669 (33)
2	bioinfo-0	Agents v1 (3-open-27-120b)	0.2359 (34)	0.1766 (23)	0.2328 (34)	0.1704 (23)
	bioinfo-1	Agents v2 (5-mixed)	0.2852 (12)	0.1527 (36)	0.2848 (12)	0.1522 (35)

Table 10: Phase B ideal answer results across all four Test Batches. Scores are shown with their overall leaderboard ranking in parentheses. The top-performing system among our submissions in each batch is highlighted in bold. Best represents the top competitor score (Rank 1); Median is the median score (and median rank) across all participants for that batch.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	E	S	Loss	Notes
LLaMA Nemotron rerank 1B	1B	2	4	multi_neg_pairwise	InfoNCE, FullData
LLaMA Nemotron rerank 1B	1B	2	—	pairwise	13B1+13B2 data
LLaMA Nemotron rerank 1B	1B	5	shifter	pairwise
BGE-reranker-v2-m3	568M	2	1	pairwise	FullData, shifter
BGE-reranker-v2-m3	568M	5	shifter	pairwise
BGE-reranker-base	278M	2	1	pairwise	FullData, shifter

Appendix

Table 11: Complete list of reranker models trained for the 14th edition. E = epochs, S = sampler, M = loss type, FullData = trained on the full dataset.

Type	Variant	Strategy
yes/no	1	Chain-of-thought: list evidence for “yes” and “no”, weigh both sides.
	2	Null-hypothesis framing: default to “no” unless evidence clearly supports “yes”.
	3	Blind verdict: neutral summary first, then decide — avoids pro/con anchoring.
	4	Claim-linked verdict: explicitly state whether abstracts support the specific claim.
factoid	1	Baseline: ranked candidate extraction, up to 5 entities.
	2	Rank-1 confidence: commit to a single best answer first, then list alternatives.

Appendix

Table 12: Exact answer prompt variants by question type. Each variant embodies a different reasoning strategy: evidence weighing, null-hypothesis framing, coverage, precision, or reformulation.

Variant	Strategy
1	Simple: context + question, answer as JSON.
2	Biomedical expert persona, chain-of-thought before output.
3	Same as 2, explicit plain-text-only constraint (no lists or markdown).
4	Same as 3, explicit concise format instruction.
5	Same as 4, includes a few-shot example (P85-Ab biomarker).
6	XML-structured context and question tags, chain-of-thought embedded in JSON.

Appendix

Table 13: Ideal answer prompt variants, ordered by complexity. All produce 50–150 word paragraph answers.

Type	Variant	Strategy
yes/no	1	Chain-of-thought, outputs both ideal and exact.
	2	Few-shot example (BRCA1), outputs both.
	3	Two-sided evidence weighing, targets macro F1 by countering yes-bias.
	4	Targets BioASQ manual evaluation criteria: recall, precision, no repetition, readability.
	5	Dual few-shot examples (one yes, one no) to anchor both answer classes equally.
	6	Mandatory evidence-quoting step before deciding.

Appendix

Table 14: Typed generation prompts: combined exact + ideal answer variants by question type.

Component	Content
System prompt	“You are a rigorous biomedical scientist participating in a structured expert debate. Your goal is to help reach the most accurate, evidence-grounded answer to a biomedical question. You respond exclusively with valid JSON.”
Document sample	Numbered documents with stable IDs, attached snippets, and adaptive retention rules (kept documents guaranteed in next round).
Conversation history	Prior turns shown in full for recent rounds, compressed to one-line summaries for older rounds to stay within context-window bounds.
Thinking focus	One of six cognitive lenses: analytical (logical decomposition), evidence-based (strict textual grounding), skeptical (gap-seeking), integrative (cross-source synthesis), pragmatic (precision-focused), or theoretical (mechanism-oriented).
Agreement scale	strongly_disagree , disagree , agree , strongly_agree — agents warned against premature strong agreement.
Output format	{"opinion": "<reasoning>", "kept_documents": [ids], "agreement": "<level>"} .

Appendix

Table 15: Agent quorum prompt components. The debate turn prompt is assembled from these sections; the final answer prompt uses a simpler synthesis-oriented structure.

Biomedical QA

Biomedical question answering develops systems that answer health and life-science questions using clinical knowledge, research literature, and structured data.

See all 51 papers

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
Jul 7, 2026Taeyun Roh, Eunha Lee, Wonjune Jang +3LLM Answer Verification Biomedical QA
Department of Computer Science and Engineering, Korea University, Seoul, 02841, Republic of Korea · Department of Mathematics, Myongji University, Yongin, 17058, Republic of Korea · 3AIGEN Sciences, Seoul, 04778, Republic of Korea
Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
May 4, 2026Devi Prasad Bal, Subhashree PuhanCross-Encoder Reranking Retrieval-Augmented Generation
The University of Texas at Austin
Retrieval Augmented Biomedical Question Answering with Weak Question Recovery and Neural Reranking for BioASQ Task 14b
Aug 2, 2026Xueying Zhao, Lee Mai, Balaji AnandganeshRetrieval-Augmented Generation Learning to Rank
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332

Jul 7, 2026J/K move · Enter open · S save

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

Taeyun Roh, Eunha Lee, Wonjune Jang, Sohyun Chung+2

Department of Computer Science and Engineering, Korea University, Seoul, 02841, Republic of Korea · Department of Mathematics, Myongji University, Yongin, 17058, Republic of Korea · 3AIGEN Sciences, Seoul, 04778, Republic of Korea

LLM Answer Verification Biomedical QA LLM Reasoning

Open paper AskPDF arXiv

Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents. This study presents a question-type-specific large language model (LLM) framework for BioASQ 14b Task B, designed to improve answer robustness and evidence grounding in biomedical question answering. Rather than applying a single prompting strategy to all questions, the framework selects different inference procedures for yes/no, factoid, and list questions according to their distinct reasoning and evaluation requirements. For yes/no questions, snippet shuffling and self-reflection are used to reduce sensitivity to evidence ordering and improve decision stability. For factoid questions, full-snippet input is combined with chain-of-thought-based in-context learning to support accurate biomedical entity identification. For list questions, a multi-agent architecture is employed, in which evidence extraction, candidate generation, answer verification, and final aggregation are handled collaboratively. Preliminary experiments on BioASQ 13b were used to identify effective inference strategies for each question type, and the resulting framework was subsequently evaluated in the official BioASQ 14b Task B challenge. In the official evaluation, our framework showed competitive performance across multiple batches and achieved first place in the factoid subtask of Batch 4. These results demonstrate the effectiveness of combining question-type-specific inference, ensemble prediction, and agent-based verification for reliable biomedical question answering.

Abstract

Biomedical QA

Explore similar work

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study

Retrieval Augmented Biomedical Question Answering with Weak Question Recovery and Neural Reranking for BioASQ Task 14b