Retrieval-Augmented Question Answering over Scientific Literature for the Electron-Ion Collider

Tina. J. Jat; T. Ghosh; Karthik Suresh

hep-exApr 2, 2026

Retrieval-Augmented Question Answering over Scientific Literature for the Electron-Ion Collider

Authors: Tina. J. Jat, T. Ghosh, Karthik Suresh

Organizations: Ramaiah University of Applied Sciences, India · College of William & Mary, USA

Abstract

To harness the power of Language Models in answering domain specific specialized technical questions, Retrieval Augmented Generation (RAG) is been used widely. In this work, we have developed a Q&A application inspired by the Retrieval Augmented Generation (RAG), which is comprised of an in-house database indexed on the arXiv articles related to the Electron-Ion Collider (EIC) experiment - one of the largest international scientific collaboration and incorporated an open-source LLaMA model for answer generation. This is an extension to it's proceeding application built on proprietary model and Cloud-hosted external knowledge-base for the EIC experiment. This locally-deployed RAG-system offers a cost-effective, resource-constraint alternative solution to build a RAG-assisted Q&A application on answering domain-specific queries in the field of experimental nuclear physics. This set-up facilitates data-privacy, avoids sending any pre-publication scientific data and information to public domain. Future improvement will expand the knowledge base to encompass heterogeneous EIC-related publications and reports and upgrade the application pipeline orchestration to the LangGraph framework.

Figures & tables

Figure 1: The schematic of the Q&A System Design for the EIC. The pipeline consists of article ingestion, document chunking, vector embedding, ChromaDB indexing, retrieval, formation of prompting to propagate the information and response generation. First, a query is encoded into a dense-vector representation during the data ingestion phase; subsequently similarity search is initiated between the embedded query and locally deployed vectorized database to retrieve the most relevant contexts. These retrieved chunks are subsequently merged with the query through a carefully designed prompt template and passed to a language model, which finally generates a contextually grounded response.

Figure 2: Left : Retrieval latency of the RAG-system for chunk size 120 and 180 characters with overlap of 20 characters between two consecutive chunks. Right: Retrieval latency measured for two two different similarity retrieval mechanism: Cosine similarity and MMR.

Figure 3: Latency distribution during answer generation of the RAG-application for two different Language Models LLaMA 3.2 and LLaMA 3.3 respectively.

Figure 4: The performance of the Q&A is represented by the six RAGAS evaluation metrics for retrieval and answer generation. Upper panel: The evaluation metrics for chunk size of 120 characters for Cosine similarity; Lower panel: Same metrics as measured for MMR retrieval technique and 120 chunk size.

Figure 5: The generation performance as characterized by the six metrics of RAGAS framework for chunk size of 180 characters with Cosine similarity(upper panel) and MMR retrieval(lower panel) respectively.

Privacy-Preserving RAG

See all 13 papers

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
Sep 29, 2026Bhagyesh Rathi, Eshan Chawla, William B. AndreopoulosLate Interaction Retrieval Scientific QA
San Jose State University, Department of Computer Science
Agentic Hybrid RAG for Evidence-Grounded Muon Collider Analysis
Jun 9, 2026Ruobing Jiang, Dawei Fu, Cheng Jiang +6Scientific QA High-Energy Physics
State Key Laboratory of Nuclear Physics and Technology, Peking University, China
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Jul 27, 2026Rajat Sainju, Dariusz Jarosz, Hairong Shang +5Graph Retrieval-Augmented Generation Scientific QA
Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois 60439, USA

Sep 29, 2026J/K move · Enter open · S save

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

Bhagyesh Rathi, Eshan Chawla, William B. Andreopoulos

San Jose State University, Department of Computer Science

Late Interaction Retrieval Scientific QA LLM Reranking Agentic Retrieval RAG Evaluation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale. In this work, we present a controlled comparison of six retrieval strategies for scientific question answering: (i) classic top-k dense retrieval, (ii) LLM-based query rephrasing, (iii) query rephrasing followed by LLM-based reranking, (iv) multi-query fusion via Reciprocal Rank Fusion (RRF), (v) an agentic tool-call pipeline in which the generator decides for itself whether to retrieve, and (vi) late-interaction retrieval with ColBERTv2. All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct), prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2 embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025. To support reproducible, large-scale evaluation, we also release a synthetic question dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains (query generation succeeded for 9,742 of them), and every strategy is evaluated on this same query set. We describe the architecture and implementation of each pipeline, release the code and the synthetic question dataset, and evaluate each strategy with an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper retrieval metrics. The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness, retrieval robustness, and agentic retrieval.

Abstract

Privacy-Preserving RAG

Explore similar work

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

Agentic Hybrid RAG for Evidence-Grounded Muon Collider Analysis

A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility