Question Answering

Also known as QA

Ask about this topic

Overview

Question answering asks how systems can find, interpret, and combine information to answer questions, from short factual queries to multi-hop reasoning over documents, conversations, images, video, and structured data. Common approaches include retrieval-augmented generation, fine-tuning, knowledge graphs, tool-using agents, and symbolic methods; evaluations span reading comprehension, open-domain and long-document QA, conversational QA, and specialized domains. Current work also focuses on multilingual and multimodal settings, personalization, and recognizing when to clarify or abstain rather than guess. A central tension is that better retrieval or reasoning does not always produce more reliable answers: systems must still handle missing or conflicting evidence, avoid hallucinations, and balance accuracy, coverage, latency, and cost.

Latest papers

CardsList

263 papers28 papers in the last four weeks, down 20% on the four weeks before. 0.2% of all new papers.

Feb 26, 2026

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.

Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian +2

IR Lab, Computer Science Department Georgetown University, Washington D.C.

Cultural Knowledge in Language ModelsCross-Cultural Language Model EvaluationAutomated EvaluationMultilingual Language Model EvaluationQuestion Answering
AskPDF
Feb 26, 2026

Towards Faithful Industrial RAG: A Reinforced Co-adaptation Framework for Advertising QA

Industrial advertising question answering (QA) is a high-stakes task in which hallucinated content, particularly fabricated URLs, can lead to financial loss, compliance violations, and legal risk. Although Retrieval-Augmented Generation (RAG) is widely adopted, deploying it in production remains challenging because industrial knowledge is inherently relational, frequently updated, and insufficiently aligned with generation objectives. We propose a reinforced co-adaptation framework that jointly optimizes retrieval and generation through two components: (1) Graph-aware Retrieval (GraphRAG), which models entity-relation structure over a high-citation knowledge subgraph for multi-hop, domain-specific evidence selection; and (2) evidence-constrained reinforcement learning via Group Relative Policy Optimization (GRPO) with multi-dimensional rewards covering faithfulness, style compliance, safety, and URL validity. Experiments on an internal advertising QA dataset show consistent gains across expert-judged dimensions including accuracy, completeness, and safety, while reducing the hallucination rate by 72%. A two-week online A/B test demonstrates a 28.6% increase in like rate, a 46.2% decrease in dislike rate, and a 92.7% reduction in URL hallucination. The system has been running in production for over half a year and has served millions of QA interactions.

Wenwei Li, Ming Xu, Tianle Xia +7

Tencent

Question AnsweringRetrieval-Augmented GenerationLLM Hallucination MitigationGraph Retrieval-Augmented GenerationRL for Language Models
AskPDF
Feb 25, 2026

LiCQA : A Lightweight Complex Question Answering System

Over the last twenty years, significant progress has been made in designing and implementing Question Answering (QA) systems. However, addressing complex questions, the answers to which are spread across multiple documents, remains a challenging problem. Recent QA systems that are designed to handle complex questions work either on the basis of knowledge graphs, or utilise contem- porary neural models that are expensive to train, in terms of both computational resources and the volume of training data required. In this paper, we present LiCQA, an unsupervised question answer- ing model that works primarily on the basis of corpus evidence. We empirically compare the effectiveness and efficiency of LiCQA with two recently presented QA systems, which are based on different underlying principles. The results of our experiments show that LiCQA significantly outperforms these two state-of-the-art systems on benchmark data with noteworthy reduction in latency.

Sourav Saha, Dwaipayan Roy, Mandar Mitra

Indian Statistical Institute, Kolkata, India · Indian Institute of Science Education and Research, Kolkata, India

Question AnsweringOpen-Domain QAMulti-Hop QA
AskPDF
Feb 22, 2026

Learning to Reason for Multi-Step Retrieval of Personal Context in Personalized Question Answering

Personalization in Question Answering (QA) requires answers that are both accurate and aligned with users' background, preferences, and historical context. Existing state-of-the-art methods primarily rely on retrieval-augmented generation (RAG) solutions that construct personal context by retrieving relevant items from the user's profile. Existing methods use the user's query directly to retrieve personal documents, and such strategies often lead to surface-level personalization. We propose PR2 (Personalized Retrieval-Augmented Reasoning), a reinforcement learning framework that integrates reasoning and retrieval from personal context for personalization. PR2 learns adaptive retrieval-reasoning policies, determining when to retrieve, what evidence to retrieve from user profiles, and how to incorporate it into intermediate reasoning steps. By optimizing multi-turn reasoning trajectories under a personalized reward function, the framework reinforces reasoning paths that better align with user-specific preferences and contextual signals reflected by the reward model. Extensive experiments on the LaMP-QA benchmark using three LLMs show that PR2 consistently outperforms strong baselines, achieving an average relative improvement of 8.8%-12% in personalized QA.

Maryam Amirizaniani, Alireza Salemi, Hamed Zamani

University of Washington Seattle, WA, USA · University of Massachusetts Amherst Amherst, MA, USA

Question AnsweringRetrieval-Augmented GenerationAgentic RAGRL for Language Model ReasoningPersonalized Language Model Alignment
AskPDF
Feb 20, 2026

RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering

Comprehensively retrieving diverse documents is crucial to address queries that admit a wide range of valid answers. We introduce retrieve-verify-retrieve (RVR), a multi-round retrieval framework designed to maximize answer coverage. Initially, a retriever takes the original query and returns a candidate document set, followed by a verifier that identifies a high-quality subset. For subsequent rounds, the query is augmented with previously verified documents to uncover answers that are not yet covered in previous rounds. RVR is effective even with off-the-shelf retrievers, and fine-tuning retrievers for our inference procedure brings further gains. Our method outperforms baselines, including agentic search approaches, achieving at least 10% relative and 3% absolute gain in complete recall percentage on a multi-answer retrieval dataset (QAMPARI). We also see consistent gains on two out-of-domain datasets (QUEST and WebQuestionsSP) across different base retrievers. Our work presents a promising iterative approach for comprehensive answer recall leveraging a verifier and adapting retrievers to a new inference scenario.

Deniz Qian, Hung-Ting Chen, Eunsol Choi

Courant Institute School of Mathematics, Computing, and Data Science, New York University

Question AnsweringEvidence Retrieval
AskPDF
Feb 20, 2026

ACE-TA: An Agentic Teaching Assistant for Grounded Q&A, Quiz Generation, and Code Tutoring

We introduce ACE-TA, the Agentic Coding and Explanations Teaching Assistant framework, that autonomously routes conceptual queries drawn from programming course material to grounded Q&A, stepwise coding guidance, and automated quiz generation using pre-trained Large Language Models (LLMs). ACE-TA consists of three coordinated modules: a retrieval grounded conceptual Q&A system that provides precise, context-aligned explanations; a quiz generator that constructs adaptive, multi-topic assessments targeting higher-order understanding; and an interactive code tutor that guides students through step-by-step reasoning with sandboxed execution and iterative feedback.

Himanshu Tripathi, Charlottee Crowell, Kaley Newlin +3

University of Alabama, Tuscaloosa, Alabama, USA · Brown University, Providence, Rhode Island, USA · Meharry Medical College, Nashville, Tennessee, USA +1

Evidence-Grounded QAIntelligent Tutoring SystemsGenerative AI in EducationQuestion Answering
AskPDF
Feb 19, 2026

Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering

Large language models (LLMs) increasingly assist software engineering tasks that require reasoning over long code contexts, yet their robustness under varying input conditions remains unclear. We conduct a systematic study of long-context code question answering using controlled ablations that test sensitivity to answer format, distractors, and context scale. Extending LongCodeBench Python dataset with new COBOL and Java question-answer sets, we evaluate state-of-the-art models under three settings: (i) shuffled multiple-choice options, (ii) open-ended questions and (iii) needle-in-a-haystack contexts containing relevant and adversarially irrelevant information. Results show substantial performance drops in both shuffled multiple-choice options and open-ended questions, and brittle behavior in the presence of irrelevant cues. Our findings highlight limitations of current long-context evaluations and provide a broader benchmark for assessing code reasoning in both legacy and modern systems.

Kishan Maharaj, Nandakishore Menon, Ashita Saxena +1

IBM Research

Long-Context Language Model EvaluationQuestion AnsweringLanguage Model Robustness
AskPDF
Feb 18, 2026

Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark

Recent advancements in Natural Language Processing and Deep Learning have enabled the development of Large Language Models (LLMs), which have significantly advanced the state-of-the-art across a wide range of tasks, including Question Answering (QA). Despite these advancements, research on LLMs has primarily targeted high-resourced languages (e.g., English), and only recently has attention shifted toward multilingual models. However, these models demonstrate a training data bias towards a small number of popular languages or rely on transfer learning from high- to under-resourced languages; this may lead to a misrepresentation of social, cultural, and historical aspects. To address this challenge, monolingual LLMs have been developed for under-resourced languages; however, their effectiveness remains less studied when compared to multilingual counterparts on language-specific tasks. In this study, we address this research gap in Greek QA by contributing: (i) DemosQA, a novel dataset, which is constructed using social media user questions and community-reviewed answers to better capture the Greek social and cultural zeitgeist; (ii) a memory-efficient LLM evaluation framework adaptable to diverse QA datasets and languages; and (iii) an extensive evaluation of 11 monolingual and multilingual LLMs on 6 human-curated Greek QA datasets using 3 different prompting strategies. We release our code and data to facilitate reproducibility.

Charalampos Mastrokostas, Nikolaos Giarelis, Nikos Karacapilidis

Industrial Management and Information Systems Lab, MEAD, University of Patras, Rio Patras, Greece

Question AnsweringMultilingual Language Model EvaluationLow-Resource Language ProcessingCultural Knowledge in Language Models
AskPDF
Feb 17, 2026

AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents

Foundation models for agriculture are increasingly trained on massive spatiotemporal data (e.g., multi-spectral remote sensing, soil grids, and field-level management logs) and achieve strong performance on forecasting and monitoring. However, these models lack language-based reasoning and interactive capabilities, limiting their usefulness in real-world agronomic workflows. Meanwhile, large language models (LLMs) excel at interpreting and generating text, but cannot directly reason over high-dimensional, heterogeneous agricultural datasets. We bridge this gap with an agentic framework for agricultural science. It provides a Python execution environment, AgriWorld, exposing unified tools for geospatial queries over field parcels, remote-sensing time-series analytics, crop growth simulation, and task-specific predictors (e.g., yield, stress, and disease risk). On top of this environment, we design a multi-turn LLM agent, Agro-Reflective, that iteratively writes code, observes execution results, and refines its analysis via an execute-observe-refine loop. We introduce AgroBench, with scalable data generation for diverse agricultural QA spanning lookups, forecasting, anomaly detection, and counterfactual "what-if" analysis. Experiments outperform text-only and direct tool-use baselines, validating execution-driven reflection for reliable agricultural reasoning.

Zhixing Zhang, Jesen Zhang, Hao Liu +4

Sun Yat-sen University

Tool-Augmented Language Model AgentsLLM AgentsScientific Reasoning in Language ModelsQuestion Answering
AskPDF
Feb 16, 2026

CGRA-DeBERTa Concept Guided Residual Augmentation Transformer for Theologically Islamic Understanding

Accurate QA over classical Islamic texts remains challenging due to domain specific semantics, long context dependencies, and concept sensitive reasoning. Therefore, a new CGRA DeBERTa, a concept guided residual domain augmentation transformer framework, is proposed that enhances theological QA over Hadith corpora. The CGRA DeBERTa builds on a customized DeBERTa transformer backbone with lightweight LoRA based adaptations and a residual concept aware gating mechanism. The customized DeBERTa embedding block learns global and positional context, while Concept Guided Residual Blocks incorporate theological priors from a curated Islamic Concept Dictionary of 12 core terms. Moreover, the Concept Gating Mechanism selectively amplifies semantically critical tokens via importance weighted attention, applying differential scaling from 1.04 to 3.00. This design preserves contextual integrity, strengthens domain-specific semantic representations, and enables accurate, efficient span extraction while maintaining computational efficiency. This paper reports the results of training CGRA using a specially constructed dataset of 42591 QA pairs from the text of Sahih alBukhari and Sahih Muslim. While BERT achieved an EM score of 75.87 and DeBERTa one of 89.77, our model scored 97.85 and thus surpassed them by 8.08 on an absolute scale, all while adding approximately 8 inference overhead due to parameter efficient gating. The qualitative evaluation noted better extraction and discrimination and theological precision. This study presents Hadith QA systems that are efficient, interpretable, and accurate and that scale provide educational materials with necessary theological nuance.

Tahir Hussain, Saddam Hussain Khan

Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering and Applied Sciences · Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering and Applied Sciences (UEAS), Swat, Pakistan · Interdisciplinary Research Center for Smart Mobility and Logistics +1

Question AnsweringKnowledge Augmentation for Language Models
AskPDF
Feb 16, 2026

The Wikidata Query Logs Dataset

We present the Wikidata Query Logs (WDQL) dataset, a dataset consisting of 335k question-query pairs over the Wikidata knowledge graph. It is over 11x larger than the largest existing Wikidata datasets of similar format without relying on template-generated queries. Instead, we construct it using real-world SPARQL queries sent to the Wikidata Query Service and generate questions for them. Since these log-based queries are anonymized, and therefore often do not produce results, a significant amount of effort is needed to convert them back into meaningful SPARQL queries. To achieve this, we present an agent-based method that iteratively de-anonymizes, cleans, and verifies queries against Wikidata while also generating corresponding natural-language questions. We demonstrate the benefit of this dataset for training question-answering methods. All WDQL assets, as well as the agent code, are publicly available via https://github.com/ad-freiburg/wikidata-query-logs under a permissive license.

Sebastian Walter, Hannah Bast

University of Freiburg, Department of Computer Science, Freiburg im Breisgau, Germany

Knowledge Graph Question AnsweringKnowledge GraphsNatural Language to SPARQLQuestion Answering
AskPDF
Jan 26, 2026

Agentic Very Long Video Understanding

The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks. Code is available at https://github.com/facebookresearch/egagent.

Aniket Rege, Arka Sadhu, Yuliang Li +5

Reality Labs Research at Meta · University of Wisconsin-Madison · Work done during internship at Meta

Question AnsweringEgocentric Video UnderstandingTool-Using Vision-Language AgentsLong-Video UnderstandingLong-Video QA
AskPDF
Jan 6, 2026

AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports

Reliable question answering over long institutional documents requires more than topical retrieval: a system must localize the exact passage that supports a claim and preserve precise numerical and temporal detail when the same indicator recurs across years, countries, and projection horizons in dense, repetitive prose. Existing question-answering benchmarks rarely test this combination of exact grounding and temporal precision at document scale. We introduce AfriEconQA, a benchmark for document-grounded question answering built from 204 World Bank economic reports on African economies, a corpus whose claims are tied to specific fiscal periods, countries, and projection states. AfriEconQA contains 4,138 evidence-linked QA instances across five reasoning categories: Factoid (1,063), List (852), Multiple Choice (889), Synthesis (671), and Comparison (663), targeting quantitative extraction, set recovery, discrimination, causal integration, and cross-period or cross-country reasoning. Each instance carries supporting evidence and source provenance and is constructed through an agentic generation pipeline with evidence-grounding checks and a stratified human-validation subset for gold-label auditing. We evaluate Qwen 3.6 35B, DeepSeek v4-pro, Gemma 4 12B IT, and frontier model GPT-5.5 under zero-shot, oracle, and hybrid retrieval-augmented generation (RAG) conditions on the held-out test split (n = 826). Across all four models, retrieval yields substantial gains, yet the best RAG system reaches only 0.594 F1, with residual errors concentrated in list extraction, synthesis, and temporally scoped comparison. AfriEconQA therefore poses a hard open challenge for exact numerical and temporal grounding over long institutional economic reports.

Edward Ajayi, Mustapha Alaba, David Stephen +1

Carnegie Mellon University Africa Kigali, Rwanda

Question AnsweringDocument QAEvidence-Grounded QATemporal QANumerical Reasoning in Language ModelsRAG Evaluation
AskPDF
Dec 14, 2025

Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM

Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA that ask for the most likely diagnosis. We introduce Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics are sourced from Dolma, the pretraining corpus for OLMo, and RedPajama, an open reproduction of the LLaMA pretraining-data recipe, FBPR achieves comparable performance to the corresponding LLMs. Furthermore, direct LLM inference and FBPR exhibit complementary strengths, correctly answering largely different subsets of questions with overlap barely above chance. Combining them with a simple hybrid method yields a notable gain over either method alone. These findings show that explicit probabilistic baselines can account for a substantial portion of the benchmark performance on diagnosis questions, while serving as historically grounded reference points and complementary signals for hybridization.

Furong Jia, Yuan Pu, Finn Guo +1

Duke University

Language Model Reasoning EvaluationClinical QAQuestion AnsweringClinical ReasoningClinical Prediction
AskPDF
Nov 30, 2025

OGD4All: A Framework for Accessible Interaction with Geospatial Open Government Data Based on Large Language Models

We present OGD4All, a transparent, auditable, and reproducible framework based on Large Language Models (LLMs) to enhance citizens' interaction with geospatial Open Government Data (OGD). The system combines semantic data retrieval, agentic reasoning for iterative code generation, and secure sandboxed execution that produces verifiable multimodal outputs. Evaluated on a 199-question benchmark covering both factual and unanswerable questions, across 430 City-of-Zurich datasets and 11 LLMs, OGD4All reaches 98% analytical correctness and 94% recall while reliably rejecting questions unsupported by available data, which minimizes hallucination risks. Statistical robustness tests, as well as expert feedback, show reliability and social relevance. The proposed approach shows how LLMs can provide explainable, multimodal access to public data, advancing trustworthy AI for open governance.

Michael Siebenmann, Javier Argota Sánchez-Vaquerizo, Stefan Arisona +3

Professorship of Computational Social Science, ETH Zurich, 8092 Zürich, Switzerland · Esri R&D Center Zurich, 8005 Zürich, Switzerland · Complexity Science Hub, Vienna, 1030, Austria

Geospatial QAQuestion AnsweringAgentic RAG
AskPDF
Oct 13, 2025

Advancing the Pareto Frontier of Test-Time Scaling with Multi-Path Plan Aggregation

Test-time scaling seeks to improve reasoning through additional inference-time computation, commonly using solution sampling with majority voting to select the final answer. However, repeated sampling can reproduce correlated errors, limiting the accuracy gains from additional computation. We introduce Multi-Path Plan Aggregation (MPPA), which learns planning and answer selection using separate LoRA adapters on a shared frozen backbone. The planner jointly generates KK short plans, each guiding a separate solution generated by the frozen executor; the aggregator then scores the resulting solutions and selects the final answer. Joint planning encourages complementary approaches suited to the executor's capabilities, while learned selection uses reasoning evidence to identify correct candidates. Alternating optimization improves planning and aggregation through supervised finetuning and reinforcement learning. Across five mathematics and science benchmarks and three backbones, MPPA improves accuracy--token trade-offs compared with training-free and training-based baselines. With three plans, it achieves higher average accuracy than eight directly sampled solutions with majority voting or the same learned aggregator, while reducing output tokens by 47--54%. Evaluation with task-provided subtasks further demonstrates higher final-task accuracy than direct sampling with majority voting or learned aggregation at either the task or subtask level.

Siheng Xiong, Ali Payani, Faramarz Fekri

Georgia Institute of Technology · Cisco Research

Test-Time ScalingLLM ReasoningMathematical ReasoningQuestion Answering
AskPDF
Jul 29, 2025

HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

Millions of individuals' well-being are challenged by the harms of substance use. Harm reduction as a public health strategy provides non-judgemental, evidence-based information intended to improve health outcomes and reduce associated safety risks. Some large language models (LLMs) have demonstrated a high level of medical reasoning, promising to address the information needs of people who use drugs (PWUD). However, their performance in relevant tasks remains largely unexplored. We introduce HarmReduction, a benchmark designed to evaluate LLMs' accuracy and safety risks in harm reduction information provision. The benchmark dataset (HR-Basic) has 2,160 question-answer-evidence pairs. The scope covers three tasks: checking safety boundaries, providing quantitative values, and inferring polysubstance use risks. We build the Instruction and RAG schemes to evaluate model behaviours based on their inherent knowledge and the integration of domain knowledge. Our results indicate that state-of-the-art LLMs still struggle to provide accurate harm reduction information, and sometimes, present severe safety risks to PWUD. This work contributes an evaluation framework for LLMs to deliver harm reduction information to avoid introducing negative health outcomes through the use of LLMs.

Kaixuan Wang, Chenxin Diao, Jason T. Jacques +2

University of Oxford, United Kingdom · University of Edinburgh, United Kingdom · University of St Andrews, United Kingdom +1

HealthcareQuestion AnsweringLLM Safety EvaluationClinical Language Model Evaluation
AskPDF
Jun 3, 2025

FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes

We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.

Christodoulos Constantinides, Dhaval Patel, Shuxin Lin +3

IBM · IBM TJ Watson Research Center

Question AnsweringLLM ReliabilityPredictive Maintenance
AskPDF
May 23, 2025

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question-answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question-answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%.

Shiyu Ni, Keping Bi, Jiafeng Guo +1

CAS Key Lab of Network Data Science and Technology, ICT, CAS · State Key Laboratory of AI Safety · University of Chinese Academy of Sciences

LLM Uncertainty EstimationLLM Hallucination MitigationFactual QAConfidence Estimation in Language ModelsHallucination in Language ModelsQuestion Answering+3
AskPDF
May 14, 2025

Atomic Consistency Preference Optimization for Long-Form Question Answering

Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accuracy by training on curated (factual, non-factual) pairs. However, this approach often relies on a stronger model (e.g., GPT-4) or an external knowledge base to assess factual correctness that may not always be accessible. Addressing this, we propose Atomic Consistency Preference Optimization (ACPO), a self-supervised preference-tuning method that enhances factual accuracy without external supervision. ACPO leverages atomic consistency signals (i.e., the agreement of individual facts across multiple stochastic responses) to identify high- and low-quality data pairs for model alignment. Despite being fully self-supervised, ACPO outperforms the strong supervised alignment baseline by 1.95 points averaged across Phi-3 and Llama3 on the LongFact and BioGen datasets, demonstrating its effectiveness in improving factual reliability without relying on external models or knowledge bases.

Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang +2

LLM Hallucination MitigationLLM AlignmentFactual Knowledge in Language ModelsPreference OptimizationQuestion AnsweringSelf-Consistency in Language Models
AskPDF
Apr 13, 2025

Kongzi: A Historical Large Language Model with Fact Enhancement

Large Language Models (LLMs) can exhibit factual inaccuracies during long-chain reasoning or under misleading prompts, limiting their reliability in historical analysis. We propose Kongzi, a language model for historical understanding and reasoning that combines continued pre-training on historical corpora, supervised fine-tuning, and fact-enhanced reinforcement learning. Its factuality reward compares entities and relations extracted from generated responses with historical references through knowledge-graph similarity. We evaluate 0.5B and 7B models on historical question answering and counterfactual reasoning, using reported ratings from three LLM judges, and on CMMLU-Chinese-History. Kongzi-7B receives the highest historical-accuracy ratings among the compared models under all three judges. Reinforcement learning improves the mean answer scores of both model sizes and their CMMLU-Chinese-History accuracy, although Kongzi-7B remains below the strongest reported baselines on that benchmark. These descriptive results support further study of factuality rewards for historical language models.

Jiashu Yang, Ningning Wang, Yian Zhao +5

Dalian University of Technology · Peking University · Nanyang Technological University

Language ModelingRL for Language Model ReasoningQuestion Answering
AskPDF
Apr 8, 2025

Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard

Despite the success of Large Language Models (LLMs) in long-context tasks, their reliability in recognizing unanswerable questions remains a critical challenge. In information-seeking scenarios, models often suffer from overconfidence and hallucinations when evidence is missing or misleading. In this paper, we propose FACTGUARD, a systematic multi-task workflow for generating evidence-grounded answerable and unanswerable questions. We introduce FACTGUARD-BENCH, a benchmark comprising 25,220 examples with context lengths up to 128K, specifically targeting Lack of Evidence and Misleading Evidence scenarios. Evaluations of eleven representative open-source and proprietary LLMs reveal a significant reliability gap: models struggle to reject unanswerable questions as context scales, with the leading model achieving only 71.62% accuracy. By training with FACTGUARD-BENCH, we improve overall accuracy to 81.17% and significantly enhance models' reasoned rejection capabilities. Our work provides a scalable path toward building more faithful and trustworthy long-context LLMs. Our source is publicly available at https://github.com/biaoliu-kiritsugu/FactGuard.

Qian-Wen Zhang, Biao Liu, Fang Li +5

Tencent YouTu Lab, Beijing, China

LLM ReliabilityQuestion AnsweringLong-Context QALong-Context Language Model Evaluation
AskPDF
Jun 15, 2024

QDA-SQL: Questions Enhanced Dialogue Augmentation for Multi-Turn Text-to-SQL

Fine-tuning large language models (LLMs) for specific domain tasks has achieved great success in Text-to-SQL tasks. However, these fine-tuned models often face challenges with multi-turn Text-to-SQL tasks caused by ambiguous or unanswerable questions. It is desired to enhance LLMs to handle multiple types of questions in multi-turn Text-to-SQL tasks. To address this, we propose a novel data augmentation method, called QDA-SQL, which generates multiple types of multi-turn Q&A pairs using LLMs. In QDA-SQL, we introduce a method incorporating validation and correction mechanisms to handle complex multi-turn Text-to-SQL tasks. Experimental results demonstrate that QDA-SQL enables fine-tuned models to exhibit higher performance on SQL statement accuracy and enhances their ability to handle complex, unanswerable questions in multi-turn Text-to-SQL tasks. The generation script and test set are released at https://github.com/mcxiaoxiao/QDA-SQL

Yinggang Sun, Ziming Guo, Haining Yu +5

School of Cyberspace Science · Harbin Institute of Technology · Harbin, China +4

Synthetic Data GenerationQuestion AnsweringText-to-SQL
AskPDF