Evidence-State Reliability Under Controlled Degradation: Parser-Validity Divergence in a Multi-Stage LLM Pipeline

Naimur Rahman

cs.CLAug 21, 2026

Evidence-State Reliability Under Controlled Degradation: Parser-Validity Divergence in a Multi-Stage LLM Pipeline

Authors: Naimur Rahman

Abstract

Multi-stage LLM pipelines can remain structurally valid even when evidence available to downstream stages becomes incomplete, compressed, or conflicting. This paper introduces and operationalizes Evidence-State Reliability (ESR), an evaluation layer concerned with whether intermediate evidence remains sufficiently complete, grounded, internally consistent, and usable for a stage's assigned function. ESR is evaluated separately from parser validity, which measures structural conformance. We evaluate the framework using GLM-5.2 on 60 sanitized base cases under four evidence conditions: clean, compressed-lossy, partial-dropout, and noisy-conflicting. Each condition was processed through decision, audit, and escalation stages. The design comprised 720 planned and ledgered calls, with 713 retained, sanitized execution rows. Across nine matched degraded-minus-clean condition-stage comparisons, all operational stage-success estimates were negative, and all 95% bootstrap intervals remained below zero. All nine parser-validity point estimates were positive, although the three partial-dropout intervals included zero. Among parser-valid degraded audit outputs, degradation detection was 1.0 in each degraded condition, while false-assurance rates remained non-zero; among parser-valid degraded escalation outputs, recovery was 0.0 in every degraded condition. The results show a bounded reliability-layer divergence in the evaluated pipeline: structural conformance can improve directionally while evidence-sensitive stage success deteriorates under the same controlled intervention. They also separate detection of degraded evidence from recovery. The conclusions are limited to the evaluated model configuration, pipeline design, selected sanitized cases, scoring procedure, and single scaled run.

Figures & tables

Measure	Value	Interpretation
Planned calls	720	$60\times 4\times 3$ design
Ledger rows	720	Complete plan-level accounting
Retained execution rows	713	Rows used for execution-level metrics
Ledger parser-valid rows	470	Structurally valid ledger outputs
Ledger parser-invalid rows	250	Includes seven ledger-only failures
Retained parser-valid rows	470	Every parser-valid ledger row retained

Table 1: Execution and parser accounting.

Condition	Stage	Retained $n$	Parser-valid $n$	Parser-valid rate	Stage-success $n$	Stage-success rate
Clean	Decision	59	30	0.508475	30	0.508475
Clean	Audit	58	30	0.517241	30	0.517241
Clean	Escalation	60	25	0.416667	25	0.416667
Compressed-lossy	Decision	60	51	0.850000	0	0.000000
Compressed-lossy	Audit	60	48	0.800000	0	0.000000
Compressed-lossy	Escalation	60	43	0.716667	0	0.000000

Table 2: Condition-by-stage structural and evidence-sensitive outcomes.

Figure 1: Paired parser-validity and evidence-sensitive stage-success changes under controlled degradation. Points show degraded-minus-clean means and 95% bootstrap intervals from 2,000 resamples with fixed seed 5205. All nine parser-validity point estimates are positive, although the three partial-dropout intervals include zero. All nine stage-success intervals remain below zero.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	+0.338983	+0.169492	+0.491525
Decision	Partial-dropout	59	+0.067797	-0.118644	+0.271186
Decision	Noisy-conflicting	57	+0.298246	+0.087719	+0.491228
Audit	Compressed-lossy	58	+0.275862	+0.086207	+0.465517
Audit	Partial-dropout	58	+0.155172	-0.034483	+0.328017
Audit	Noisy-conflicting	57	+0.368421	+0.210526	+0.526316

Table 3: Paired degraded-minus-clean changes in parser validity.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	-0.508475	-0.644068	-0.372881
Decision	Partial-dropout	59	-0.508475	-0.644068	-0.372881
Decision	Noisy-conflicting	57	-0.491228	-0.614035	-0.368421
Audit	Compressed-lossy	58	-0.517241	-0.637931	-0.396121
Audit	Partial-dropout	58	-0.517241	-0.637931	-0.396121
Audit	Noisy-conflicting	57	-0.508772	-0.631579	-0.385965

Table 4: Paired degraded-minus-clean changes in operational stage success.

Stage	Degraded condition	Paired $n$	Mean $\Delta$	95% low	95% high
Decision	Compressed-lossy	59	-0.508475	-0.644068	-0.372881
Decision	Partial-dropout	59	-0.508475	-0.644068	-0.372881
Decision	Noisy-conflicting	57	-0.473684	-0.596491	-0.350877
Audit	Compressed-lossy	58	-0.517241	-0.637931	-0.396121
Audit	Partial-dropout	58	-0.517241	-0.637931	-0.396121
Audit	Noisy-conflicting	57	-0.508772	-0.631579	-0.385965

Table 5: Paired degraded-minus-clean changes in the model-output-coded evidence-state-adequacy indicator.

Figure 2: Audit detection and escalation recovery under controlled evidence degradation. Among parser-valid degraded outputs, the audit degradation-detection rate was 1.0 in each degraded condition, while escalation recovery was 0.0 in every degraded condition. These are model-output-coded indicators from the implemented computational pipeline and do not represent institutional audit effectiveness.

Figure 3: Sequence-level cascade-failure rate across all evidence conditions. The operational cascade definition classified 223 of 240 condition-linked sequence groups as failures. This aggregate includes both clean and degraded conditions and should not be interpreted as a degradation treatment effect or as an estimate of failure prevalence in deployed systems.

Condition	Groups	Complete	Failures	Failure rate	Pattern composition
Clean	60	57	43	0.716667	40 parser failure; 3 incomplete; 17 preserved success
Compressed-lossy	60	60	60	1.000000	29 parser failure; 29 detected but not recovered; 2 audit false assurance
Partial-dropout	60	60	60	1.000000	46 parser failure; 14 detected but not recovered
Noisy-conflicting	60	57	60	1.000000	28 parser failure; 28 detected but not recovered; 1 audit false assurance; 3 incomplete
All conditions	240	234	223	0.929167	143 parser failure; 71 detected but not recovered; 3 audit false assurance; 6 incomplete; 17 preserved success

Table 6: Sequence-level reliability-cascade composition.

Figure 4: Sequence-level failure-family composition. Across 240 condition-linked groups, the operational taxonomy contained 143 parser-failure cascades, 71 detected-but-unrecovered patterns, three audit-false-assurance patterns, six incomplete sequences, and 17 preserved successes.

Explore similar work

Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures
Mar 17, 2026Oleg Somov, Mikhail Chaichuk, Gleb Ershov +4LLM Reasoning Causal Interventions in Language Models
AIRI · MIPT · HSE University +2
TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis
Aug 25, 2026Boshen Shi, Yize Liu, Chen Zhao +4LLM Reliability LLM Evaluation
China Mobile Jiutian Artificial Intelligence Technology (Beijing) Co., Ltd. · School of Computer and Cyberspace Security, Communication University of China
PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines
May 12, 2026Varun KotteConformal Prediction
Independent Researcher.

Mar 17, 2026J/K move · Enter open · S save

Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

Oleg Somov, Mikhail Chaichuk, Gleb Ershov, Karim Vafin+3

AIRI · MIPT · HSE University+2

LLM Reasoning Causal Interventions in Language Models Reasoning Consistency in Language Models

Open paper AskPDF arXiv

In schema-guided reasoning (SGR) pipelines, LLMs produce explicit intermediate structures -- rubrics, checklists, or verification queries -- before committing to a final decision. SGR is increasingly adopted because it promises controllability: practitioners expect to inspect, edit, and override these structures to steer the outcome. But does the promise hold? We introduce a causal evaluation protocol to measure it: by selecting tasks where a deterministic function maps intermediate structures to decisions, every controlled edit implies a unique correct output. Across 12 models and 4 benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention -- revealing that apparent faithfulness is fragile once the intermediate structure changes. When derivation of the final decision from the structure is delegated to an external tool, this fragility largely disappears; stronger prompting yields only limited improvements, while preference optimization substantially improves intervention faithfulness. Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators.

Abstract

Explore similar work

Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines