Evidence-State Reliability Under Controlled Degradation: Parser-Validity Divergence in a Multi-Stage LLM Pipeline

Naimur Rahman

cs.CLAug 21, 2026

Evidence-State Reliability Under Controlled Degradation: Parser-Validity Divergence in a Multi-Stage LLM Pipeline

Authors: Naimur Rahman

Abstract

Multi-stage LLM pipelines can remain structurally valid even when evidence available to downstream stages becomes incomplete, compressed, or conflicting. This paper introduces and operationalizes Evidence-State Reliability (ESR), an evaluation layer concerned with whether intermediate evidence remains sufficiently complete, grounded, internally consistent, and usable for a stage's assigned function. ESR is evaluated separately from parser validity, which measures structural conformance. We evaluate the framework using GLM-5.2 on 60 sanitized base cases under four evidence conditions: clean, compressed-lossy, partial-dropout, and noisy-conflicting. Each condition was processed through decision, audit, and escalation stages. The design comprised 720 planned and ledgered calls, with 713 retained, sanitized execution rows. Across nine matched degraded-minus-clean condition-stage comparisons, all operational stage-success estimates were negative, and all 95% bootstrap intervals remained below zero. All nine parser-validity point estimates were positive, although the three partial-dropout intervals included zero. Among parser-valid degraded audit outputs, degradation detection was 1.0 in each degraded condition, while false-assurance rates remained non-zero; among parser-valid degraded escalation outputs, recovery was 0.0 in every degraded condition. The results show a bounded reliability-layer divergence in the evaluated pipeline: structural conformance can improve directionally while evidence-sensitive stage success deteriorates under the same controlled intervention. They also separate detection of degraded evidence from recovery. The conclusions are limited to the evaluated model configuration, pipeline design, selected sanitized cases, scoring procedure, and single scaled run.

Figures & tables

Measure	Value	Interpretation
Planned calls	720	$60\times 4\times 3$ design
Ledger rows	720	Complete plan-level accounting
Retained execution rows	713	Rows used for execution-level metrics
Ledger parser-valid rows	470	Structurally valid ledger outputs
Ledger parser-invalid rows	250	Includes seven ledger-only failures
Retained parser-valid rows	470	Every parser-valid ledger row retained

Table 1: Execution and parser accounting.

Condition	Stage	Retained $n$	Parser-valid $n$	Parser-valid rate	Stage-success $n$	Stage-success rate
Clean	Decision	59	30	0.508475	30	0.508475
Clean	Audit	58	30	0.517241	30	0.517241
Clean	Escalation	60	25	0.416667	25	0.416667
Compressed-lossy	Decision	60	51	0.850000	0	0.000000
Compressed-lossy	Audit	60	48	0.800000	0	0.000000
Compressed-lossy	Escalation	60	43	0.716667	0	0.000000

Table 2: Condition-by-stage structural and evidence-sensitive outcomes.

Figure 1: Paired parser-validity and evidence-sensitive stage-success changes under controlled degradation. Points show degraded-minus-clean means and 95% bootstrap intervals from 2,000 resamples with fixed seed 5205. All nine parser-validity point estimates are positive, although the three partial-dropout intervals include zero. All nine stage-success intervals remain below zero.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	+0.338983	+0.169492	+0.491525
Decision	Partial-dropout	59	+0.067797	-0.118644	+0.271186
Decision	Noisy-conflicting	57	+0.298246	+0.087719	+0.491228
Audit	Compressed-lossy	58	+0.275862	+0.086207	+0.465517
Audit	Partial-dropout	58	+0.155172	-0.034483	+0.328017
Audit	Noisy-conflicting	57	+0.368421	+0.210526	+0.526316

Table 3: Paired degraded-minus-clean changes in parser validity.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	-0.508475	-0.644068	-0.372881
Decision	Partial-dropout	59	-0.508475	-0.644068	-0.372881
Decision	Noisy-conflicting	57	-0.491228	-0.614035	-0.368421
Audit	Compressed-lossy	58	-0.517241	-0.637931	-0.396121
Audit	Partial-dropout	58	-0.517241	-0.637931	-0.396121
Audit	Noisy-conflicting	57	-0.508772	-0.631579	-0.385965

Table 4: Paired degraded-minus-clean changes in operational stage success.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	-0.508475	-0.644068	-0.372881
Decision	Partial-dropout	59	-0.508475	-0.644068	-0.372881
Decision	Noisy-conflicting	57	-0.473684	-0.596491	-0.350877
Audit	Compressed-lossy	58	-0.517241	-0.637931	-0.396121
Audit	Partial-dropout	58	-0.517241	-0.637931	-0.396121
Audit	Noisy-conflicting	57	-0.508772	-0.631579	-0.385965

Table 5: Paired degraded-minus-clean changes in the model-output-coded evidence-state-adequacy indicator.

Figure 2: Audit detection and escalation recovery under controlled evidence degradation. Among parser-valid degraded outputs, the audit degradation-detection rate was 1.0 in each degraded condition, while escalation recovery was 0.0 in every degraded condition. These are model-output-coded indicators from the implemented computational pipeline and do not represent institutional audit effectiveness.

Figure 3: Sequence-level cascade-failure rate across all evidence conditions. The operational cascade definition classified 223 of 240 condition-linked sequence groups as failures. This aggregate includes both clean and degraded conditions and should not be interpreted as a degradation treatment effect or as an estimate of failure prevalence in deployed systems.

Condition	Groups	Complete	Failures	Failure rate	Pattern composition
Clean	60	57	43	0.716667	40 parser failure; 3 incomplete; 17 preserved success
Compressed-lossy	60	60	60	1.000000	29 parser failure; 29 detected but not recovered; 2 audit false assurance
Partial-dropout	60	60	60	1.000000	46 parser failure; 14 detected but not recovered
Noisy-conflicting	60	57	60	1.000000	28 parser failure; 28 detected but not recovered; 1 audit false assurance; 3 incomplete
All conditions	240	234	223	0.929167	143 parser failure; 71 detected but not recovered; 3 audit false assurance; 6 incomplete; 17 preserved success

Table 6: Sequence-level reliability-cascade composition.

Figure 4: Sequence-level failure-family composition. Across 240 condition-linked groups, the operational taxonomy contained 143 parser-failure cascades, 71 detected-but-unrecovered patterns, three audit-false-assurance patterns, six incomplete sequences, and 17 preserved successes.

Explore similar work

Mar 17, 2026

Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

In schema-guided reasoning (SGR) pipelines, LLMs produce explicit intermediate structures -- rubrics, checklists, or verification queries -- before committing to a final decision. SGR is increasingly adopted because it promises controllability: practitioners expect to inspect, edit, and override these structures to steer the outcome. But does the promise hold? We introduce a causal evaluation protocol to measure it: by selecting tasks where a deterministic function maps intermediate structures to decisions, every controlled edit implies a unique correct output. Across 12 models and 4 benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention -- revealing that apparent faithfulness is fragile once the intermediate structure changes. When derivation of the final decision from the structure is delegated to an external tool, this fragility largely disappears; stronger prompting yields only limited improvements, while preference optimization substantially improves intervention faithfulness. Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators.

Oleg Somov, Mikhail Chaichuk, Gleb Ershov +4

AIRI · MIPT · HSE University +2

LLM Reasoning Causal Interventions in Language Models Reasoning Consistency in Language Models

Ask PDF

Aug 25, 2026

TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.

Boshen Shi, Yize Liu, Chen Zhao +4

China Mobile Jiutian Artificial Intelligence Technology (Beijing) Co., Ltd. · School of Computer and Cyberspace Security, Communication University of China

LLM Reliability LLM Evaluation Language Model Robustness

Ask PDF

May 12, 2026

PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines

Modern NLP and LLM systems are pipelines: named entity recognition (NER) -> entity disambiguation (NED) -> entity typing, retrieval-augmented generation (retriever -> reader), and agentic chains of planner -> tool -> critic. Errors compound across stages, but existing uncertainty quantification methods either calibrate each stage independently (no joint coverage) or apply a Bonferroni union bound (joint coverage, but conservative). We present PASC (Pipeline-Aware Split Conformal), which reduces multi-stage joint coverage to a single scalar conformal prediction problem on the joint maximum nonconformity score. PASC provides a finite-sample distribution-free guarantee that all K stages are simultaneously covered with probability at least 1 - alpha, and is nearly tight up to a 1/(n+1) factor. On a three-stage NER -> NED -> entity-typing pipeline over CoNLL-2003, PASC achieves 96.4% end-to-end coverage versus 93.4% for Bonferroni and 86.5% for independent CP, at identical average prediction set size (1.083). Under distribution shift to WNUT-17 Twitter and WikiNEuRal Wikipedia data, PASC empirically maintains the target coverage in the tested shift settings while independent CP collapses to 59%. PASC requires a single quantile computation, runs 1.7x faster than Bonferroni, and scales to K = 6 stages where independent CP drops to 0.53 end-to-end coverage. The same joint-maximum-score reduction applies directly to compound LLM systems and agent pipelines.

Varun Kotte

Independent Researcher.

Conformal Prediction

Ask PDF