LLM Auditing

LLM: Large Language Model

Overview

Large language model auditing examines how models are trained and how they behave in use, looking for problems such as bias, privacy leakage, hallucinations, unsafe actions, and unreliable reasoning. Common approaches include controlled benchmarks and counterfactual tests, multi-turn red-teaming, membership-inference and provenance analysis, human review, and probes of model activations or weights. Audits increasingly examine whole systems—including prompts, tools, interfaces, and model updates—rather than judging only a model’s final answer. A central challenge is that results can shift with prompts, access methods, versions, and evaluators, while behavioral scores or internal signals alone may not establish why a failure occurred or whether it will recur in deployment.

Latest papers

723 papers79 papers in the last four weeks, down 15% on the four weeks before. 0.6% of all new papers.

Feb 17, 2026

A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Models

Large language models and LLM-based agents are increasingly used for cybersecurity tasks that are inherently dual-use. Existing approaches to refusal, spanning academic policy frameworks and commercially deployed systems, often rely on broad topic-based bans or offensive-focused taxonomies. As a result, they can yield inconsistent decisions, over-restrict legitimate defenders, and behave brittlely under obfuscation or request segmentation. We argue that effective refusal requires explicitly modeling the trade-off between offensive risk and defensive benefit, rather than relying solely on intent or offensive classification. In this paper, we introduce a content-based framework for designing and auditing cyber refusal policies that makes offense-defense tradeoffs explicit. The framework characterizes requests along five dimensions: Offensive Action Contribution, Offensive Risk, Technical Complexity, Defensive Benefit, and Expected Frequency for Legitimate Users, grounded in the technical substance of the request rather than stated intent. We demonstrate that this content-grounded approach resolves inconsistencies in current frontier model behavior and allows organizations to construct tunable, risk-aware refusal policies.
Feb 17, 2026

Discovering Implicit Large Language Model Alignment Objectives

Large language model (LLM) alignment relies on complex reward signals that often obscure the specific behaviors being incentivized, creating critical risks of misalignment and reward hacking. Existing interpretation methods typically rely on pre-defined rubrics, risking the omission of "unknown unknowns", or fail to identify objectives that comprehensively cover and are causal to the model behavior. To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives. Our approach utilizes an iterative greedy algorithm to analyze behavioral changes across training checkpoints, identifying and validating candidate objectives that best explain the residual reward signal. Extensive evaluations across diverse tasks, model sizes, and alignment algorithms demonstrate the framework's robustness. Experiments with popular open-source reward models show that the framework consistently captures > 90% of reward behavior, a finding further corroborated by human evaluation. Additionally, a case study on alignment with an open-source reward model reveals that Obj-Disco can successfully identify latent misaligned incentives that emerge alongside intended behaviors. Our work provides a crucial tool for uncovering the implicit objectives in LLM alignment, paving the way for more transparent and safer AI development.
Feb 16, 2026

Automatically Finding Reward Model Biases

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.
Feb 16, 2026

Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

Large Language Models (LLMs) are increasingly deployed in contact-center Quality Assurance (QA) to automate agent performance evaluation and coaching feedback. While LLMs offer unprecedented scalability and speed, their reliance on web-scale training data raises concerns regarding demographic and behavioral biases that may distort workforce assessment. We present a counterfactual fairness evaluation of LLM-based QA systems across 13 dimensions spanning three categories: Identity, Context, and Behavioral Style. Fairness is quantified using the Counterfactual Flip Rate (CFR), the frequency of binary judgment reversals, and the Mean Absolute Score Difference (MASD), the average shift in coaching or confidence scores across counterfactual pairs. Evaluating 18 LLMs on 3,000 real-world contact center transcripts, we find systematic disparities, with CFR ranging from 5.4% to 13.0% and consistent MASD shifts across confidence, positive, and improvement scores. Larger, more strongly aligned models show lower unfairness, though fairness does not track accuracy. Contextual priming of historical performance induces the most severe degradations (CFR up to 16.4%), while implicit linguistic identity cues remain a persistent bias source. Finally, we analyze the efficacy of fairness-aware prompting, finding that explicit instructions yield only modest improvements in evaluative consistency. Our findings underscore the need for standardized fairness auditing pipelines prior to deploying LLMs in high-stakes workforce evaluation.
Feb 15, 2026

NEST: Nascent Encoded Steganographic Thoughts

Monitoring chain-of-thought (CoT) reasoning is a foundational safety technique for large language model agents; however, this oversight is compromised if models learn to conceal their reasoning. We explore steganographic CoT--where models hide secret reasoning within innocuous text--to inform risk assessment and deployment policies. Steganographic reasoning requires two skills in a single forward pass: computing an intermediate result, and embedding it into a coherent cover that answers an unrelated question. Drawing on our taxonomy of steganographic and non-steganographic CoT types, we systematically evaluate the limits of prompt-elicited steganographic CoT capability across 34 models, ranging from past generations to the current frontier. We measure monitor evasion, refusal rates, encoding fidelity, and hidden task accuracy across five datasets, comparing against plain reasoning, direct answer, and filler-token baselines. The two experiments isolate the two sub-skills: a reasoning tasks sweep tests joint reason-and-embed, while a counting task hands the model a known numerical sequence and tests embedding alone--a necessary precondition for stego reasoning. Current frontier models cannot sustain joint reason-and-embed: a paired McNemar comparison shows the steganographic channel is dominated by an filler-token baseline on every (model, family) cell. The encoding-only floor, by contrast, is cleared--Claude Opus~4.5 reaches 92% per-number partial accuracy on 4-digit sequences and saturates at 100% exact-match on length-8 single-digit sequences--establishing that the binding constraint on stego CoT is the joint reasoning-plus-encoding load, not raw channel capacity. Our findings underscore the need for continuous evaluation of steganographic risk and provide a methodology to preemptively detect and evaluate hidden reasoning that might empower misaligned scheming and deceptive behavior.
Feb 12, 2026

Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs

Large Language Models (LLMs) are advancing rapidly, yet the benchmarks used to measure this progress are becoming increasingly unreliable. Score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy. We introduce the Benchmark Health Index (BHI), a pure data-driven framework for auditing evaluation sets along three orthogonal and complementary axes: (1) Capability Discrimination, measuring how sharply a benchmark separates model performance beyond noise; (2) Anti-Saturation, estimating remaining headroom before ceiling effects erode resolution and thus the benchmark's expected longevity; and (3) Impact, quantifying influence across academic and industrial ecosystems via adoption breadth and practice-shaping power. By distilling 106 validated benchmarks from the technical reports of 91 representative models in 2025, we systematically characterize the evaluation landscape. BHI is the first framework to quantify benchmark health at a macro level, providing a principled basis for benchmark selection and enabling dynamic lifecycle management for next-generation evaluation protocols.
Feb 10, 2026

Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

Multi-agent systems (MAS) can substantially extend the reasoning capacity of large language models (LLMs). Most MAS frameworks aggregate agent outputs via simple majority voting, discarding the evidential structure of reasoning traces. Majority voting is brittle under confabulation consensus, where agents share correlated biases and converge on the same incorrect rationale. We introduce AgentAuditor, which moves beyond frequency-based aggregation by organizing agent traces into a Reasoning Tree that explicitly represents agreements and divergences in their reasoning. AgentAuditor resolves conflicts by comparing branch-level evidence at critical divergence points, turning global adjudication into efficient, localized verification. We further propose Anti-Consensus Preference Optimization (ACPO), which trains the adjudicator with evidence-verified preference supervision to reduce conformity to misleading majority cues. Across four MAS frameworks and multiple reasoning benchmarks, AgentAuditor consistently improves aggregation performance over majority voting, with gains of up to 5% absolute accuracy while remaining token-efficient.
Feb 3, 2026

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

Chain-of-thought (CoT) prompting is widely used as a reasoning aid and is often treated as a transparency mechanism. Yet behavioral gains under CoT do not imply that the model's internal computation causally depends on the emitted reasoning text, i.e. models may produce fluent rationales while routing decision-critical computation through latent pathways. We introduce a causal, layerwise audit of CoT faithfulness based on activation patching. Our key metric, the CoT Mediation Index (CMI), isolates CoT-specific causal influence by comparing performance degradation from patching CoT-token hidden states against matched control patches. Across multiple model families (Phi, Qwen, DialoGPT) and scales, we find that CoT-specific influence is typically depth-localized into narrow ''reasoning windows,'' and we identify bypass regimes where CMI is near-zero despite plausible CoT text. We further observe that models tuned explicitly for reasoning tend to exhibit stronger and more structured mediation than larger untuned counterparts, while Mixture-of-Experts models show more distributed mediation consistent with routing-based computation. Overall, our results show that CoT faithfulness varies substantially across models and tasks and cannot be inferred from behavior alone, motivating causal, layerwise audits when using CoT as a transparency signal.
Feb 2, 2026

Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

Large-scale foundation models exhibit behavioral shifts when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning. Current explainability methods are structurally ill-suited to explain these shifts, because they either treat models as static objects, as traditional eXplainable AI (XAI) approaches do, or merely compare independent explanations across different checkpoints of a model. As a result, these approaches fail to explain the functional transition between two model instances in which a certain behavior has shifted following an intervention. This gap creates significant governance risks across jurisdictions including the EU AI Act, US state legislation, and Chinese AI regulations, which require documenting causal chains for substantial system modifications. This position paper argues that explaining behavioral shifts in large language models requires a principled approach that treats the shift itself as the primary object of explanation: namely, one that explains how and why an intervention transforms a reference model into an updated model with different behavior. To support this claim, we introduce Comparative XAI (XAIΔ_Δ), a novel XAI paradigm aimed at explaining the difference between two model checkpoints where a behavior has shifted, together with a set of desiderata specifying what XAIΔ_Δ explainers and explanations must satisfy, including comparability, validity, actionability, and monitoring, with the goal of grounding model auditing in explicit, measurable requirements. Finally, we provide preliminary evidence suggesting the need for XAIΔ_Δ in practice through illustrative experiments, compiling the resulting findings into a transition report directly usable for governance and incident documentation.
Jan 30, 2026

Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines

Large Language Models (LLMs) are increasingly deployed in tool-driven environments such as healthcare analytics, financial systems, retrieval-augmented generation (RAG), and multi-agent workflows. Although the Model Context Protocol (MCP) standardizes how LLM applications expose and invoke external tools, its baseline model does not require tool-use manifests to be cryptographically authenticated, freshness-checked, policy-bound, or independently auditable before execution. As a result, MCP pipelines may remain vulnerable to manifest tampering, unauthorized tool invocation, replay of stale requests, and weak accountability. This paper presents a manifest-level enforcement layer for MCP-based LLM pipelines. It treats each MCP tool-use manifest as a first-class security object whose canonical form must be policy-validated, freshness-checked, digitally signed, verified before execution, and linked to tamper-evident audit evidence. The framework binds tool invocation to verifiable manifest integrity and fail-closed authorization, separates user-visible request parameters from execution metadata, rejects non-compliant or stale manifests before execution, and records accepted invocations in a Merkle-based transparency log. Evaluation across GPT-5.3, LLaMA-3.5, and DeepSeek-V3 using up to 50,000 manifest instances shows near-linear scalability (R^2 = 0.998), bounded verification latency (<= 9.4 ms on edge devices), and rejection of expired, malformed, replayed, and policy-violating manifests, with rejection rates above 98.7%. Experiments in healthcare, finance, RAG, and multi-agent settings show that manifest-level cryptographic enforcement enables low-overhead, traceable, and auditable execution control for heterogeneous LLM-tool pipelines.
Jan 29, 2026

Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas

Language models are increasingly consulted on ethically consequential questions, yet the stance a model expresses may not survive a change in framing. We audit 16 models across 14 ethically fraught dilemmas using polarity-paired proposals ("They should X" / "They should not X"). A model's judgment of the underlying action should not reverse merely because the question is phrased as a prohibition rather than a prescription and yet, we find systematic deviations from this invariance including wholesale endorsement flips, indicating that ethical decisions are vulnerable to framing instability. Small open-weight models (1-4B parameters) endorse a proposed action 24% of the time under affirmative framing but up to 100% under negated framings, a swing of as much as 76 percentage points. Human coding of a response sample confirms the instability is genuine while showing that binary agree/disagree proxies over-state its magnitude, suggesting that an LLM judge cannot replace human coders because it silently collapses abstentions and mirrors the very forced-choice bias under study. Commercial models are for the most part more stable but still shift substantially, with cross-model agreement dropping from 73% on the bare affirmative framing to 59% under simple negation. We argue that because binary agree/disagree formats both inflate apparent endorsement and mask polarity-dependence, single-phrasing audits can misreport a model's ethical stance, and we propose the Negation Sensitivity Index (NSI) as a complement that measures stance stability directly. A model whose stance flips with phrasing cannot be relied upon in any high-stakes decision scenario.
Jan 26, 2026

Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection?

LLMs expand accessibility and provide wide-reaching access to information. Yet these interactions also create opportunities to embed subtle, goal-oriented behaviours that shape what users think and how they behave, a concern reflected in governance frameworks that prohibit manipulative AI. We refer to these behaviours as hidden intentions: covert agendas embedded in a model's outputs that can manipulate users' beliefs and actions. In this work, we examine whether hidden intentions can be identified and characterised, and assess whether detection can serve as a mitigation strategy. To operationalise this, we introduce a social-science-grounded set of ten hidden intention categories and show that they are trivially inducible. A case study further confirms that all ten categories manifest in deployed LLMs. We then evaluate static classifiers and LLM judges on these categories, providing the first systematic analysis of why hidden intentions are difficult to detect. Our stress tests show that, unless false-positive rates are vanishingly small, auditing is dominated by precision-prevalence trade-offs. Capability scaling and reasoning models do not close this gap, suggesting a fundamental challenge for open-world detection. These findings expose a core gap of current AI governance: without new auditing paradigms for open-world, low-prevalence risks, bans on manipulative AI remain difficult to enforce.
Jan 23, 2026

White-Box Sensitivity Auditing with Steering Vectors

Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently indicates substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit
Jan 20, 2026

Pro-AI Bias in Large Language Models

Large language models (LLMs) are increasingly employed for decision-support across multiple domains. We ask whether these models favor artificial intelligence (AI) itself. We examine this question through three experiments. First, in open-ended advice across investment, study, career, and startup domains, AI is prominent in most recommendation lists and is usually ranked first when it appears. Second, in salary estimations, models overestimate AI-related jobs more than they do for closely matched non-AI jobs. In a controlled follow-up, changing only the job title from the matched non-AI title to the AI-related title raises predicted salary by 10%, while all other supplied job details remain fixed. Finally, we show that the number of degrees and other credentials awarded by AI programs were 28% more overestimated than corresponding awards for matched computing programs. Overall, we show that AI is not only prominent in everyday LLM advice, but also receives systematically inflated quantitative estimates compared to non-AI cases, even against its closest disciplinary neighbors, raising concerns for high-stakes decision support.
Jan 18, 2026

A Scalable Entity-Based Framework for Auditing Bias in Large Language Models

Existing approaches to bias evaluation in large language models (LLMs) trade ecological validity for statistical control, relying either on artificial prompts that poorly reflect real-world use or on naturalistic tasks that lack scale and rigor. We introduce a scalable bias-auditing framework that uses named entities as controlled probes to measure systematic disparities in model behavior. Synthetic data enables us to construct diverse, controlled inputs, and we show that it reliably reproduces bias patterns observed in natural text, supporting its use for large-scale analysis. Using this framework, we conduct the largest bias audit to date, comprising 1.9 billion data points across multiple entity types, tasks, languages, models, and prompting strategies. We find consistent patterns: models penalize right-wing politicians and favor left-wing politicians, prefer Western and wealthier countries over the Global South, favor Western companies, and penalize firms in the defense and pharmaceutical sectors. While instruction tuning reduces bias, increasing model scale amplifies it, and prompting in Chinese or Russian does not mitigate Western-aligned preferences. These findings highlight the need for systematic bias auditing before deploying LLMs in high-stakes applications. Our framework is extensible to other domains and tasks, and we make it publicly available to support future work.
Jan 7, 2026

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

Large language models (LLMs) can generate fluent peer reviews but often miss sparse, subtle, and document-level scientific errors. We introduce PaperAudit-Bench, a benchmark for evaluating paper-level error detection and error-aware automated review. It comprises (1) PaperAudit-Dataset, 220 high-quality AI/ML papers containing more than 2,200 controlled synthetic errors across eight categories, and (2) PaperAudit-Review, an evidence-aware framework that links structured error detection to review generation. Experiments across models, error categories, paper sections, and analysis depths show that sparse long-context auditing remains challenging for current LLMs. Conditioning review generation on explicit, manuscript-grounded error evidence produces more conservative and discriminative assessments that align more closely with human judgments than representative automated-review baselines. The dataset also supports post-training compact detectors through SFT and RL; Qwen3-14B achieves 55.8% F1 at Detection@1 under a fixed held-out protocol. A PubPeer transfer study and deployment in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences, further illustrate practical applicability. Code and Dataset Page: https://github.com/TU2021/PaperAudit ScienceOne Manuscript Review: https://www.scienceone.cn/lit/#/assistant
Jan 6, 2026

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g., ΔΔ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40×\times fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at ε=0.02\varepsilon=0.02 for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.
Dec 3, 2025

Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we derive policy-violation scores directly from normalized representations of LLM hidden activations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks. It achieves up to 86.0% F1, outperforming both fine-tuned and LLM-as-a-judge baselines while requiring substantially less computation. Our code is publicly available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection
Nov 28, 2025

MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS

Large Language Model (LLM)-based Multi-Agent Systems (MAS) are susceptible to linguistic attacks that can trigger cascading failures across the network. Existing defenses face a fundamental dilemma: lightweight single-auditor methods are prone to single points of failure, while robust committee-based approaches incur prohibitive computational costs in multi-turn interactions. To address this challenge, we propose MAS-Shield, a secure and efficient defense framework designed with a coarse-to-fine filtering pipeline. Rather than applying uniform scrutiny, MAS-Shield dynamically allocates defense resources through a three-stage protocol: (1) Critical Agent Selection strategically targets high-influence nodes to narrow the defense surface; (2) Light Auditing employs lightweight sentry models to rapidly filter the majority of benign cases; and (3) Global Consensus Auditing escalates only suspicious or ambiguous signals to a heavyweight committee for definitive arbitration. This hierarchical design effectively optimizes the security-efficiency trade-off. Experiments demonstrate that MAS-Shield achieves a 92.5% recovery rate against diverse adversarial scenarios and reduces defense latency by over 70% compared to existing methods.
Oct 13, 2025

Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniqueness

Auditing information disclosure across every datapoint during the training of LLMs is challenging. We propose a principled, attack-agnostic approach that uses mutual information to measure what the final model reveals about a datapoint's training membership. We show that this final-model disclosure is upper bounded by the sum of per-iteration gradient disclosures and that, under a reasonable set of assumptions, these gradient disclosures increase with Gradient Uniqueness (GNQ), which measures how distinguishable a datapoint's gradient is relative to other gradients in the batch. While naively computing GNQ requires forming and inverting a P×PP\times P matrix for every datapoint (for a model with PP parameters), we introduce Batch-Space Ghost (BS-Ghost). This efficient algorithm performs all computations in a much smaller batch space and uses ghost kernels to compute GNQ "in-run" for every datapoint in the training corpus, with minimal computational and memory overhead. Our experiments show the following: (i) GNQ predicts MIA vulnerability without the need for shadow models. (ii) Beyond membership disclosure, GNQ predicts the success of reconstruction attacks. (iii) GNQ-guided removal and retraining identify datapoints that causally contribute to disclosure. (iv) GNQ outperforms counterfactual memorization in text extraction and common-knowledge discrimination without the need for additional model training. (v) For data attribution, GNQ-guided filtering reduces emergent misalignment in Qwen2.5-7B more than baselines. Further, GNQ attributes 1000 datapoints in 17 seconds---roughly 70×70\times faster than the baselines. (vi) GNQ explains how training choices affect training-set disclosure and captures how per-datapoint disclosure emerges during training.
Oct 8, 2025

Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5

LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.
Oct 7, 2025

The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge. While Inverse Reinforcement Learning (IRL) can infer reward functions from behaviour, existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task (non-identifiability). This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification. Our framework leverages Bayesian IRL to not only recover a distribution over objectives but to enable three critical audit capabilities: (i) Quantifying and systematically reducing non-identifiability by demonstrating posterior contraction over sequential rounds of evidence; (ii) Providing actionable, uncertainty-aware diagnostics that expose spurious shortcuts and identify out-of-distribution prompts where the inferred objective cannot be trusted; and (iii) Validating policy-level utility by showing that the refined, low-uncertainty reward can be used directly in RLHF to achieve training dynamics and toxicity reductions comparable to the ground-truth alignment process. Empirically, our framework successfully audits a detoxified LLM, yielding a well-calibrated and interpretable objective that strengthens alignment guarantees. Overall, this work provides a practical toolkit for auditors, safety teams, and regulators to verify what LLMs are truly trying to achieve, moving us toward more trustworthy and accountable AI.
Oct 7, 2025

Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

We propose a system for marking sensitive or copyrighted texts to detect their use in fine-tuning large language models under black-box access with statistical guarantees. Our method builds digital marks'' using invisible Unicode characters organized into (cue'', reply'') pairs. During an audit, prompts containing only cue'' fragments are issued to trigger regurgitation of the corresponding ``reply'', indicating document usage. To control false positives, we compare against held-out counterfactual marks and apply a ranking test, yielding a verifiable bound on the false positive rate. Empirically, we obtain a true positive rate of 96.7% at 0% false positive rate and reply regurgitation rates exceeding 28% per document with only 40 (4%) watermarked documents. The approach is minimally invasive, scalable across many sources, robust to standard processing pipelines, and achieves high detection power even when marked data is a small fraction of the fine-tuning corpus.
Sep 27, 2025

WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

Benchmarks built from arXiv papers can overlap the arXiv text used in LLM pretraining, yet technical-domain releases rarely ship the per-item evidence needed to check this. We introduce a reverse-probe n-gram audit that indexes the benchmark side and streams the corpus past it, so memory scales with the benchmark rather than the corpus and flags on the distributed problems can be inspected and recomputed. We instantiate it in WirelessMathBench-XL, a 4,027-problem benchmark for wireless mathematical reasoning built from 836 arXiv papers across 20 subfields, with multiple-choice, progressive fill-in, and full-equation-completion formats. Streaming 12.9 B RedPajama-arXiv 13-grams yields a strict zero-hit view S0\mathcal{S}_0 covering 95.7% of the problems, and shows that the Dolma and ProofPile-2 arXiv slices repackage the same RedPajama dump. Filtering to S0\mathcal{S}_0 moves every evaluated model by less than 1 pp. With 30/800 flagged test items, this comparison bounds detected-overlap inflation at 0.31-0.51 pp for frontier models rather than testing for contamination, and it does not cover paraphrase, target-answer, alternate-version, post-training, or closed-corpus exposure. Frontier models reach 86.5-91.3%, and GRPO on the released training split lifts a 7B model 25.1 pp above a size-matched general-math control. We release source-paper identifiers, per-problem audit and threshold metadata, license-aware data files, audit and bootstrap scripts, Croissant metadata, and a Datasheet.
Sep 26, 2025

What Is The Political Content in LLMs' Pre- and Post-Training Data?

Large language models (LLMs) reflect politically-slanted opinions in their generated text. Even though it is widely assumed that model behavior stem from training data, there has been no study quantifying the extent to which political content is part of the training data. To bridge this gap, we aim to directly estimate (1)~the proportion of politically engaged texts in training data, (2)~respective data imbalance, (3)~cross-dataset similarity, and (4)~correlations between data composition and model behaviour. We analyze the political content of pre- and post-training datasets of open-source LLMs, combining large-scale sampling, political-leaning classification, and stance detection. We find that all LLM training datasets are systematically skewed towards left-leaning content, with pre-training containing more politically engaged than post-training corpora. We further observe a strong correlation between political stances in training data and model behavior, which is present already in most base models and persists across post-training stages. These findings highlight the role of data composition in correlating with model behavior and motivate the need for greater data transparency as a means to understand and monitor model behavior.
Sep 22, 2025

The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

Large language models (LLMs) are increasingly used to simulate human collective behavior, yet claims that such simulations are human-like remain largely untested. We conducted a systematic audit (pre-registered on OSF) of LLM-based social simulations across four databases (Scopus, IEEE Xplore, ACM Digital Library, and arXiv). Across 576 studies reported in 350 recent papers, we applied six methodological evaluations: agent Profile, Interaction, Memory, Minimal-Control, Unawareness, and Realism (PIMMUR). Coding every study against pre-specified rules, we revealed that PIM were met more often than MUR. Frontier LLMs correctly identified the underlying social experiment in 65.2% of cases, and 50.6% of prompts imposed constraints that pre-determined the outcome. These compliance rates are upper bounds, because incomplete methodological reporting (for example, unreleased prompts) limits the available evidence. Reproducing five representative experiments (e.g., opinion dynamics), we found that reported collective phenomena often vanish or reverse once PIMMUR principles are enforced, indicating that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Current LLM simulations may therefore capture model-specific biases rather than universal features of human social behavior, raising concerns about their use as scientific proxies for human society.
Aug 25, 2025

Speculating LLMs' Chinese Training Data Pollution from Their Tokens

Tokens are basic elements in the datasets for LLM training. It is well-known that many tokens representing Chinese phrases in the vocabulary of GPT (4o/4o-mini/o1/o3/4.5/4.1/o4-mini) are indicating contents like pornography or online gambling. Based on this observation, our goal is to locate Polluted Chinese (PoC) tokens in LLMs and study the relationship between PoC tokens' existence and training data. (1) We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary. (2) We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies by considering each token's both semantics and related contents from the search engines. (3) We study the speculation on the training data pollution via PoC tokens' appearances (token ID). Experiments on GPT and other 23 LLMs indicate that tokens widely exist while GPT's vocabulary behaves the worst: more than 40% long Chinese tokens (i.e., a token with more than two Chinese characters) are either porn or online gambling. We validate the accuracy of our speculation method on famous pre-training datasets like C4 and Pile. Then, considering GPT-4o, we speculate that the ratio of "Yui Hatano" related webpages in GPT-4o's training data is around 0.5%.
Jul 30, 2025

Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming

Large language models (LLMs) are increasingly used to answer health-related questions and support healthcare workflows, yet evidence for their safety still relies heavily on static benchmarks that can rapidly become obsolete or be optimized against. Here we introduce a Dynamic, Automatic, and Systematic (DAS) red-teaming audit framework that continuously stress-tests LLMs for health across four safety-critical axes: robustness, privacy, bias/fairness, and hallucination/factual inaccuracies. Validated against board-certified clinicians with high concordance, a suite of adversarial agents autonomously mutates health-related test cases to uncover vulnerabilities in real time. Applying DAS to 15 proprietary and open-source LLMs revealed a profound gap between high static benchmark performance and low dynamic reliability--the "Benchmarking Gap". Despite median MedQA accuracy exceeding 80%, 94% of previously correct answers failed under dynamic robustness testing. This brittleness generalized to the realistic, open-ended HealthBench dataset, where top-tier models exhibited failure rates exceeding 70% and sharp shifts in model rankings across evaluations, suggesting that high scores on established static benchmarks may reflect superficial memorization. We observed similarly high failure rates across other domains: privacy leaks were elicited in 86% of scenarios, cognitive-bias priming altered recommendations in 81% of fairness tests, and hallucination rates exceeded 74% in widely used models. By converting LLM safety evaluation for health from a static checklist into a living adversarial audit, DAS provides a scalable framework for surfacing latent risks before such systems are deployed in consumer-facing health assistants, clinician-facing tools, and broader healthcare workflows. Code is available at https://github.com/JZPeterPan/DAS-Medical-Red-Teaming-Agents.
Jun 19, 2025

PRISON: Unmasking the Criminal Potential of Large Language Models

As large language models (LLMs) advance, concerns about their misconduct in complex social contexts intensify. Existing research overlooked the systematic understanding and assessment of their criminal capability in realistic interactions. We propose a unified framework PRISON, to quantify LLMs' criminal potential across five traits: False Statements, Frame-Up, Psychological Manipulation, Emotional Disguise, and Moral Disengagement. Using structured crime scenarios adapted from classic films grounded in reality, we evaluate both criminal potential and anti-crime ability of LLMs. Results show that state-of-the-art LLMs frequently exhibit emergent criminal tendencies, such as proposing misleading statements or evasion tactics, even without explicit instructions. Moreover, when placed in a detective role, models recognize deceptive behavior with only 44% accuracy on average, revealing a striking mismatch between conducting and detecting criminal behavior. These findings underscore the urgent need for adversarial robustness, behavioral alignment, and safety mechanisms before broader LLM deployment.
May 23, 2025

But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

LLM-as-a-judge is widely used as a scalable substitute for human evaluation, yet judges are least reliable on subtle dishonesty such as sycophancy and manipulation, where a judge scoring a response in isolation lacks a reference point. We introduce Judge Using Safety-Steered Alternatives (JUSSA), a framework that reduces this evaluator uncertainty using the assessed model's own internal representations. An honesty-promoting steering vector, optimized from a single training example, generates a contrastive alternative against which each response is judged. On a new manipulation dataset with human-validated response pairs, JUSSA improves AUROC for both GPT-4.1 (0.893→\rightarrow0.946) and Claude Haiku (0.859→\rightarrow0.929) judges. The gains concentrate almost entirely on the pairs a judge would otherwise misrank. They reverse once a judge has saturated a task and for the weakest judge we test, so contrastive evaluation helps most when a judge is capable enough to use the reference but has not yet saturated the task. A layer-wise analysis further locates steering effectiveness in the middle layers, where model representations of honest and dishonest prompt processing diverge. Our work demonstrates that steering vectors can serve as tools for evaluation rather than for improving model outputs at inference, opening a new direction for white-box auditing.