LLM Answer Verification

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to check an LLM’s answers, reasoning steps, citations, and evidence before accepting, revising, or abstaining from them. Approaches include retrieval and natural-language inference for evidence checking, symbolic tools and proof assistants for formal tasks, learned reward models for ranking candidate answers, and self- or cross-model review. These methods are tested in settings such as mathematical reasoning, multi-hop question answering, long-document and medical QA, and code or SQL generation, using benchmarks including MATH, GPQA, HotpotQA, FEVER, and BIRD. A central challenge is that verifiers can share a generator’s blind spots or reject valid answers, so accuracy must be balanced against calibration, evidence coverage, cost, and the risk of harmful revisions.

Latest papers

CardsList

151 papers11 papers in the last four weeks, down 63% on the four weeks before. 0.1% of all new papers.

Mar 3, 2025

HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs

An Achilles heel of Large Language Models (LLMs) is their tendency to hallucinate non-factual statements. A response mixed of factual and non-factual statements poses a challenge for humans to verify and accurately base their decisions on. To combat this problem, we propose Highlighted Chain-of-Thought Prompting (HoT), a technique for prompting LLMs to generate responses with XML tags that ground facts to those provided in the question. That is, given an input question, LLMs would first re-format the question to add XML tags highlighting key facts, and then, generate a response with highlights over the facts referenced from the input. Compared to vanilla chain of thought prompting (CoT), HoT reduces the rate of hallucination and separately improves LLM accuracy consistently on over 22 tasks from arithmetic, reading comprehension, to logical reasoning. When asking humans to verify LLM responses, highlights help time-limited participants to more accurately and efficiently recognize when LLMs are correct. Yet, surprisingly, when LLMs are wrong, HoTs tend to fool users into believing that an answer is correct.

Tin Nguyen, Logan Bolton, Mohammad Reza Taesiri +2

Auburn University · University of Alberta

LLM GroundingLLM Answer VerificationLLM Hallucination MitigationCoT ReasoningHallucination in Language ModelsLLM Reliability
AskPDF