A framework for auditing grounding claims

Daniel Quigley; Eric Maynard

cs.AIDec 5, 2025

A framework for auditing grounding claims

Authors: Daniel Quigley, Eric Maynard

Organizations: Center for Possible Minds Indiana University Bloomington Bloomington, IN 47408 · Eruditis Milwaukee, WI 53207

Abstract

The symbol grounding problem asks how a token such as cat can be about cats. We propose a framework for auditing grounding claims against a declared semantic standard. The audit reports measurements and evidence, with overall verdicts conditional on explicit acceptance criteria. Its profiles assess accuracy, robustness, and composition alongside evidence about how the system acquired its mechanisms, how they contribute to performance, and why they were retained. In a toy gridworld, an agent interprets individual symbols accurately but fails a withheld combination. Composing its interpretations by the declared rule would succeed. This comparison identifies a departure from the composition rule within the observed failure. Both this audit and a pilot on pretrained word vectors provide evidence that a designated mechanism contributes to present performance. Whether that contribution explains its retention remains uncertified. The framework evaluates the evidence for grounding claims; candidate accounts remain responsible for explaining how meaning emerges.

Figures & tables

Clause	Question	Comparison
G0 authenticity	Do the mechanisms that carry symbols to meanings belong to the system, and did its own learning or evolution produce them?	inspection of provenance for each component relied upon, relative to the declared boundary and history: membership, then acquisition
G1 preservation	Does each atom mean what the standard says it means?	the realized meaning of each atom against its target, aggregated over the atoms
G2a correlational faithfulness	Does each composite mean what the standard says it means?	the realized meaning of each composite against its target, aggregated over the composites
G2b etiological faithfulness	Does the designated mechanism contribute to success now, and was it retained for that contribution?	success with the mechanism on against success with it off, on the decoupled distribution; historical controls for the retention
G3 robustness	How far does realized meaning move under a declared perturbation?	the realized meaning of an item against the realized meaning of its perturbed self, as a curve over scales
G4 compositionality	Does a composite mean what the algebra makes of the system’s own parts, and do withheld combinations pass?	the realized composite against the algebra applied to the realized atoms; the pass rate over withheld composites

Table 1: Diagnostic clauses

Panel of Figure 1 — Figure 1: The grounding architecture encodes surface symbols into internal representations and reads them out into a meaning space; its maps belong to the agent level $L_{\text{ag}}$ . The evaluation index, the intended interpretation, and the profile belong to the auditor level $L_{\text{aud}}$ ; the meaning space is the interface shared by both, drawn at the boundary of the architecture.

Clause	Measurement	Declarations and assessment	Acceptance
G0	component membership and acquisition records	declare system boundary, history $\mathcal{T}$ , and components relied upon; inspect their provenance. Record instrument components separately	weak: system membership; strong: own learning or evolution
G1	atomic discrepancy $\varepsilon_{\text{pres}}$	declare target, metric, atomic population, and aggregate (maximum or quantile)	$\varepsilon_{\text{pres}}\leq\bar{\varepsilon}_{\text{pres}}$
G2a	composite discrepancy $\varepsilon_{\text{faith}}$	declare target, metric, composite population, and aggregate (maximum or quantile).	$\varepsilon_{\text{faith}}\leq\bar{\varepsilon}_{\text{faith}}$
G2b	causal effect $\text{ACE}(M)$ and historical certification status	declare succ , $M$ , on/off intervention, and construction of $P^{\text{do}}$ from $P$ ; assess retention using stated historical controls and selection assumptions.	$\text{ACE}(M)\geq\eta$ and historical requirement satisfied
G3	robustness curve $\omega_{U}(\varepsilon)$ : $(1-\alpha)$ -quantile over $P$ of worst-case semantic change at scale $\varepsilon$	declare metric, population $P$ , perturbation location ( $\mathcal{R}$ or $\Sigma$ ), threat family, scales, and coverage $\alpha$ ; report supremum or approximation used; sampled perturbation quantiles estimate the worst case from below	$\omega_{U}(\varepsilon)\leq\bar{\omega}(\varepsilon)$ over declared range
G4	compositional deviation $\delta_{\text{comp}}$ and withheld-composite accuracy $\beta$	declare metric, grammar, algebra, constructor applications, and deviation aggregate; for $\beta$ , declare targets, withheld population, its overlap with G2a, and item tolerance $\tau$ .	$\delta_{\text{comp}}\leq\bar{\delta}_{\text{comp}}$ and $\beta\geq\bar{\beta}$

Table 2: Audit measurements, declarations, and acceptance conditions.

Clause	Result	Population or scope
G0	strong: system components acquired through training	embedding, GRU, and readout
G1	maximum discrepancy $0.098$	six atoms
G2a	maximum discrepancy $0.580$ ; success rate $0.875$	eight composites, including both withheld commands
G2b	$\text{ACE}(M)=0.875\geq\eta=0.10$ ; historical status uncertified	directional integration ablated; eight composites under $P^{\text{do}}=P$
G3	hidden-state supremum $0.840$ ; sampled directional mean $0.095$	radius $0.5$ ; mean over $256$ sampled directions
G4: deviation	maximum $0.062$ on trained pairs; $0.670$ on withheld pairs	six trained and two withheld constructor applications

Table 3: Grid-world profile. Measurements are descriptive unless an acceptance condition is stated.

Format or coupling	Representative realization	Audit	Questions for measurement
Symbolic	discrete expressions with lookup or rule-based interpretation; simplest case, $\mathcal{R}=\Sigma$ and $\Phi=\operatorname{id}$	logical context, inferential meanings, surface-edit threats	atomic and composite accuracy; agreement with semantic algebra; sensitivity to edits; acquisition and retention of rules
Vectorial	$\mathcal{R}=\mathbb{R}^{n}$ with declared metric, an encoder $\Phi$ , readout $\mathcal{A}$	linguistic or visual contexts, with threats specified at input or representation level	target agreement, geometric alignment where declared, perturbation sensitivity, composition on novel combinations
Relational	typed nodes, edges, or axioms, with graph metric and readout based on roles or rule closure	inferential meanings under edits to relations, types, or axioms	agreement with target relations; propagation of edits through inference; composition across typed structures
Referential coupling	sensorimotor representations connected to world or task states through perception and action	extensional meanings in embodied context, under declared environmental perturbations	accuracy of world-directed interpretation, causal dependence on coupling, evidence of retention for task success

Table 4: Representation formats and referential coupling; realizations illustrate possible audit specifications, and grounding verdicts come from the audits themselves.

Clause	Stipulated formal interpreter	Human language use
G0	weak when designed interpretation is implemented within declared system	strong where audited mechanisms were acquired within a history including evolution and development
G1	zero discrepancy on atoms in interpretation’s domain	agreement depends on context, sense resolution, and declared standard
G2a	zero discrepancy on well-typed composites interpreted by same semantic algebra	agreement must be assessed across declared constructions and communicative situations
G2b	historical requirement fails in stipulated case, whose construction excludes selection for task success	acquisition and communicative feedback motivate selection hypotheses; certification requires mechanism- and task-specific evidence
G3	every edit below minimum spacing is trivial; behavior at larger scales depends on interpretation and treatment of malformed inputs	tolerance of benign variation is expected locally; its extent depends on threat and population
G4	zero compositional deviation; any nonempty valid withheld set has $\beta=1$ where $g=\mathcal{I}$	productive interpretation motivates systematicity tests; idioms and coercion require an appropriate grammar and algebra

Table 5: Reference profiles for a stipulated formal interpreter and ordinary human language use. The human column states expectations and evidential requirements; its entries are unmeasured.

	G2b certified	G2b uncertified	G2b failed
G2a passes	target agreement with etiological support	target agreement; etiology unresolved	target agreement despite established etiological deficit
G2a fails	target error despite etiological support	target error; etiology unresolved	target error and established etiological deficit

Table 6: Correlational and etiological outcomes under a fixed audit specification. Uncertified G2b records insufficient evidence to establish satisfaction or failure.

Pattern	Profile	Diagnostic question
Exact composition, inaccurate atoms	G1 fails while $\delta_{\text{comp}}=0$ . Composite accuracy and $\beta$ require separate assessment	Does changing the atomic interpretation improve target agreement while preserving composition? Atomic errors may cancel or accumulate.
Accurate composites, divergent composition	G2a passes while $\delta_{\text{comp}}>\bar{\delta}_{\text{comp}}$ on evaluated applications.	Is the declared algebra appropriate? If so, what mechanism supports accuracy despite departure from it?
Familiar accuracy, withheld failure	G2a passes on familiar composites, but $\beta<\bar{\beta}$ on strict withheld population.	Does failure involve inaccurate constituents, departure from the algebra, or both? Memorization is one hypothesis among several.
Accuracy with perturbation sensitivity	G2a passes on clean composites while G3 fails under declared threat and bound.	Where does the perturbation enter, and which mechanism amplifies its effect? Front-end correction and readout control address different sources.
Instrument-dependent agreement	Accuracy is measured through an auditor-supplied readout or composer, recorded as instrument under G0.	Does the system implement a corresponding interpretation or operation? Accuracy measured through an adapter is credited to the instrument.

Table 7: Diagnostic patterns under declared audit conditions. Each row specifies only the coordinates relevant to the contrast; the others remain open. Suggested tests are diagnostic; whether a repair works is settled by evaluation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Clause	Instrument	Quantity	Value
G0	inspection	atomic $\Phi$ : system, own learning	strong
		composite extension; readout $\mathcal{A}$	instrument
G1	MEN, evaluation pairs	median atomic discrepancy; fraction within $0.20$	$0.138$ ; $77.9\%$
	SimLex, evaluation pairs	median atomic discrepancy; fraction within $0.20$	$0.186$ ; $54.9\%$
G2a	WordNet tiers, all adjectives	mean Kendall discrepancy; perfect-order rate	$0.193$ ; $42.1\%$
G2b	permutation ablation, MEN	$\widehat{\text{ACE}}$ [bootstrap $95\%$ interval]; $\eta=0.10$	$0.292$ $[0.283,0.301]$ ; estimate above $\eta$ ; history uncertified

Appendix

Table 8: Profile of pruned word2vec. Populations are identified by row. Thresholds used for comparisons were fixed before inspection; undeclared acceptance bounds leave measurements descriptive. G3 reports sampled perturbation quantiles.

Figure 2: Sampled perturbation quantiles

\hat{q}_{U}

, using one draw per retained item at each scale. Typo results depend on the edit and exclusion policies. The curves are descriptive with respect to

\omega_{U}

Panel of Figure 3 — Figure 3: RED WEST in the grid world. The sum of the realized atoms (open square) lies $0.115$ from the target (filled circle), inside the success radius. The realized composite (filled triangle) lies $0.670$ from that sum and $0.580$ from the target, outside the radius.

Explore similar work

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntAgentic Evaluations Llm-As-A-Judge
Independent Researcher
GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs
Apr 25, 2026Federico A. KamelharMulti-Agent Large Language Model Systems Llm-As-A-Judge
Senior Principal Architect, Agentic AI Oracle Corporation
Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems
Apr 29, 2026Mahnoor Shahid, Hannes RotheNeuro-Symbolic Framework Compositional Generalization
Universität Duisburg Essen, Germany

Jun 22, 2026, cs.AIJ/K move · Enter open · S save

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Jeffrey Flynt

Independent Researcher

Agentic Evaluations Llm-As-A-Judge Tool-Grounded Reasoning Physical Plausibility

Open paper AskPDF arXiv

Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.

Abstract

Explore similar work

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs

Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems