Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

Jing Liu; Yongxing Qi; Ping Yao; Ting Wu

cs.AIOct 13, 2026

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

Authors: Jing Liu, Yongxing Qi, Ping Yao, Ting Wu

Organizations: Hangzhou Innovation Institute, Beihang University, Hangzhou, China · Unit, Beijing, China

Abstract

What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the public TCMEval-SDT benchmark for Traditional Chinese Medicine syndrome differentiation (200 cases), the binding constraint is output discipline and reasoning structure, not model capability alone: two years of model scaling move zero-shot scores by only -0.03 to +0.08, while output-discipline failures (non-terminating reasoning, format non-compliance, large run-to-run spread) persist even in frontier models. We test this with a model-agnostic agent framework formalizing decision making as a five-stage closed loop, enforced by a candidate x evidence matrix falsification protocol: every candidate is judged against every evidence item, at least half of the candidates must be falsified before a conclusion is licensed, and survivors are emitted in a machine-verifiable single-line format. Across five base models, the effect is sign-conditional on zero-shot output discipline: it rescues models whose discipline fails (DeepSeek-V4-Flash: +0.267/+0.271, entirely termination rescue), reduces a high-variance model's run-to-run standard deviation from 0.218 to 0.007 (Kimi K3), and significantly costs already-disciplined models (-0.087 to -0.120 on budget-matched Qwen3-4B/32B). Ablations attribute the cost to the candidate-bounded format layer and show the effect is independent of injected knowledge content (protocol-knowledge decoupling). The protocol also transfers to misinformation-robust mathematical reasoning, detecting injected false premises at 89.8% recall with a 6.8% false-positive rate (MCC 0.54; a flag-everything baseline scores 0) and restoring accuracy to the no-misinformation ceiling. Where zero-shot output discipline is weak, the reasoning protocol sets the floor; base-model capability always sets the ceiling.

Figures & tables

Figure 1: Four-layer architecture. The epistemic layer defines the hypothesis–verification–causal-analysis–action loop; the TCM process layer carries the differentiation workflow, closing the loop through the validation-feedback stage; the model layer (LLM-1/LLM-2/ML/RAG) executes it; the data layer holds syndrome–sign mappings, reasoning data, case mappings, and safety rules. Within the process layer, verification takes the form of targeted questioning over typical sign–syndrome data, and causal analysis proceeds over the five-phase self-consistent system.

Figure 2: End-to-end processing pipeline (S1–S9) with anti-deadlock questioning guards, the insufficiency re-examination loop after matrix falsification, and prescription-violation interception.

Figure 3: The candidate

\times

four-examination evidence matrix falsification protocol on a worked example. Three knowledge inputs build the matrix; row-level aggregation yields compatibility verdicts; the sufficiency check requires at least half of the candidates to be incompatible, otherwise evidence is re-examined; the most-supported compatible candidate is emitted in the single-line protocol format.

Method stage	Lifecycle state(s)
Intent triage, direct retrieval	outside the machine
Information collection, sufficiency	COLLECT
Hypothesis & evidence construction	DIAGNOSE (prompt injection)
Matrix falsification & output	DIAGNOSE (output protocol)
Causal-chain reasoning	DIAGNOSE (reasoning)
Action: plan, prescribe, review	PLAN $\rightarrow$ PRESCRIBE $\rightarrow$ REVIEW

Table 1: Mapping between method stages and lifecycle states.

Figure 4: Official-script scores on TCMEval-SDT (our own runs, n=200 per task). Paired bars show zero-shot vs. +agent for each base model; error bars mark the standard deviation over three repetitions where available. Dashed lines mark the benchmark paper’s best reported zero-shot baselines (2023–2024 era, 50-case subset). Note the K3 pair: the zero-shot error bar (

\pm 0.218

, descriptive spread over the

n{=}3

runs) versus the agent’s (

\pm 0.007

) is the visual evidence for the stabilization claim—the protocol reduces run-to-run standard deviation by an order of magnitude (point estimate 31-fold).

Model	Variant	Syndrome	Pathogen.
Qwen3-4B	zero-shot	0.481 $\pm$ 0.005	0.455 $\pm$ 0.010
Qwen3-4B	+ agent	0.365 $\pm$ 0.013	0.335 $\pm$ 0.010
PT-4B	zero-shot	0.473 ‡	0.457 ‡
PT-4B	+ agent	0.362 ‡	0.345 ‡
Qwen3-32B	zero-shot	0.551 $\pm$ 0.007	0.523 $\pm$ 0.006
Qwen3-32B	+ agent	0.464 $\pm$ 0.011	0.411 $\pm$ 0.009

Table 2: Official-script scores on TCMEval-SDT (our own runs, n=200 per task). Values are mean

\pm

std over 3 repetitions;

\dagger

marks single runs; ‡ marks single budget-matched greedy runs (8,192-token budget). Paper baselines are 2023–2024-era zero-shot on a 50-case subset, converted to per-case means. Bold values mark per-column maxima; the bold row label marks the protocol configuration.

		Syndrome			Pathogen.
Model	Variant	EM	F1	hit	EM	F1	hit
Qwen3-4B	zero	20.0	0.588	87.0	9.5	0.578	92.5
Qwen3-4B	agent	16.5	0.465	70.0	7.0	0.425	72.5
PT-4B	zero	17.5	0.589	89.0	8.0	0.585	92.5
PT-4B	agent	15.0	0.444	67.0	8.5	0.452	77.0
Qwen3-32B	zero	29.7	0.644	88.0	16.7	0.636	92.0

Table 3: Auxiliary metrics (n=200 per task, C2/C3 campaigns as in Table 2 ): exact match (EM, %), macro-F1, and set hit rate (%). The hit–score gap quantifies the over-prediction failure mode of Section 5.7 : high hit with a modest official-script score indicates over-selection (e.g., 4B zero-shot: 87.0% hit vs. 0.481 score).

Model	Task	Answer rate (%)	Over-pred.	Run std	Cross-run agr. (%)	$\Delta$
DS-Flash	syn.	38.5	0.42	0.015	60.0	$+0.267$
DS-Flash	pat.	33.0	0.42	0.012	62.5	$+0.271$
DS-Pro	syn.	62.5	0.75	—	—	$+0.094$
DS-Pro	pat.	51.0	0.66	—	—	$+0.210$
K3	syn.	99.5	1.38	0.218	10.5	$+0.080$
K3	pat.	97.0	1.43	0.043	37.0	$-0.032$

Table 4: Output-discipline profile of each base model, measured on protocol-free zero-shot outputs (200 cases per task), against the protocol’s gain

\Delta

(agent

-

zero-shot; CIs and

p

-values in Table 5 ). Answer-extraction rate : fraction of cases whose output yields a machine-extractable answer under the benchmark’s official extraction rule, computed on the first repetition of each arm (DS-Pro’s only repetition), so the rate is comparable across arms—for thinking models this coincides with termination within budget, and it subsumes format non-compliance. Over-prediction ratio : per-case mean of

|\mathrm{pred}|/|\mathrm{gold}|

. Run std : standard deviation of run means across three repetitions. Cross-run agreement : fraction of cases with identical predicted label sets across all three runs. Run std and self-consistency require three repetitions and are unavailable for the single-run zero-shot configuration (DS-Pro).

Figure 5: The protocol’s gain against the zero-shot answer-extraction rate, per base model and task. Each model the protocol helps carries a distinct measurable failure mode (non-termination for flash, partial non-termination for Pro, instability for K3); the two models it hurts show none. Rank correlations are reported qualitatively at

n{=}5

Model	Task	$\Delta$	95% CI	$p$ (case)	$p$ (run)
Qwen3-4B	syndrome	$-$ 0.116	[ $-$ 0.167, $-$ 0.064]	2.6e-05	1.7e-03
Qwen3-4B	pathogen.	$-$ 0.120	[ $-$ 0.157, $-$ 0.084]	8.1e-10	1.0e-04
DS-Flash	syndrome	+0.267	[+0.226, +0.307]	6.1e-22	1.3e-02
DS-Flash	pathogen.	+0.271	[+0.234, +0.307]	2.0e-24	6.7e-02
DS-Pro	syndrome	+0.094	[+0.039, +0.149]	4.2e-03	—
DS-Pro	pathogen.	+0.210	[+0.131, +0.295]	1.4e-12	—

Table 5: Paired tests of the mechanism’s gain (zero-shot

\rightarrow

+agent) per base model and task: per-case mean difference

\Delta

with bootstrap 95% CI and Wilcoxon

p

(n=200 cases). Raw

p

-values; the family-wise Bonferroni threshold for this 10-test family is 0.005. The Qwen3-4B comparison is budget-matched (both sides greedy with an 8,192-token budget, three repetitions each on the same vLLM backend). DS-Pro rows are single-repetition (case-level tests only) and are descriptive under the two-level standard of Section 4.4 . The 4B negative result cannot be attributed to truncation. Run-level

p

: conservative replication check—Welch’s

t

-test over the three run means per side (Section 4.4 ), uncorrected; “—” for configurations with a single-run side.

Model	Subset	$n$	Zero	Agent	$\Delta$ (95% CI)	$p$
DS-Flash syn.	full	200	0.289	0.556	$+0.267$ [ $+0.226,+0.307$ ]	6.1e-22
	reliably terminating	43	0.866	0.781	$-0.085$ [ $-0.133,-0.037$ ]	2.0e-03
	rescue-prone	157	0.132	0.494	$+0.363$ [ $+0.325,+0.401$ ]	2e-25
DS-Flash pat.	full	200	0.212	0.482	$+0.271$ [ $+0.233,+0.308$ ]	2e-24
	reliably terminating	32	0.709	0.622	$-0.087$ [ $-0.140,-0.034$ ]	3.1e-03
	rescue-prone	168	0.117	0.456	$+0.339$ [ $+0.304,+0.374$ ]	4e-27

Table 6: Rescue/terminating decomposition of the zero-shot

\rightarrow

agent gain on the DeepSeek models (paired per-case differences, 95% CI, Wilcoxon

p

). Reliably terminating : zero-shot yields an extractable answer in every repetition (DS-Pro: single run)—a strictly stronger condition than the per-repetition answer-extraction rate of Table 4 , which is why the flash subset (43/200 cases) is smaller than that rate (38.5%) suggests. Rescue-prone : the complement. K3 and the Qwen models terminate at

\geq

97% zero-shot answer rate, so for them the full-set contrast in Table 5 is already a terminating-case estimate. For DS-Pro the agent-side budget is 16,384 tokens and roughly half of its terminating case-runs exceed the 8k-equivalent line, so its rescue-prone gains retain a budget component that the post-hoc proxy (Appendix F ) can only partially exclude. DS-Pro rows are single-repetition and hence descriptive under the two-level standard of Section 4.4 . Full-set rows duplicate Table 5 for readability.

Arm (content / matrix / format)	Syn.	Pat.	Hit syn./pat. (%)
zero-shot ( $-$ / $-$ / $-$ )	0.481 $\pm$ 0.005	0.455 $\pm$ 0.010	87.5 / 92.0
format only ( $-$ / $-$ /+)	0.240	0.172	38.0 / 38.0
content only (G1, +/ $-$ / $-$ )	0.388 $\pm$ 0.009	0.319 $\pm$ 0.007	59.0 / 66.7
no matrix (G5, +/ $-$ /+)	0.392 $\pm$ 0.007	0.343 $\pm$ 0.004	67.3 / 72.8
no format (G6, +/+/ $-$ )	0.334 $\pm$ 0.007	0.284 $\pm$ 0.002	67.5 / 70.5
full protocol (+/+/+)	0.365 $\pm$ 0.013	0.335 $\pm$ 0.010	67.5 / 74.2

Table 7: The completed three-factor factorial on budget-matched Qwen3-4B (three repetitions per arm, n=200 per task). The content bundle is knowledge injection + causal chain; “format” is the candidate-bounded single-line emission mandate; “matrix” is the falsification-matrix requirement.

Group	Syn.	Pat.	Syn.	Pat.	Syn.	Pat.
	Official-script score		$\Delta$ vs. G0 (CI, $p$ )		Hit $\Delta$ (pp)
G0 full agent	0.365 $\pm$ 0.013	0.335 $\pm$ 0.010	—	—	67.5	74.2
G1 $-$ matrix+format block	0.388 $\pm$ 0.009	0.319 $\pm$ 0.007	+0.022 (n.s.)	$-0.015$ (n.s.)	$-8.5$	$-7.5$
G2 $-$ evidence checklist	0.403 $\pm$ 0.009	0.332 $\pm$ 0.004	+0.038 (n.s.)	$-0.003$ (n.s.)	+6.2	$-1.6$
G3 $-$ diff. guide	0.391 $\pm$ 0.018	0.347 $\pm$ 0.015	+0.026 (n.s.)	+0.012 (n.s.)	+2.2	$-0.7$
G4 $-$ all skill knowledge	0.364 $\pm$ 0.009	0.353 $\pm$ 0.004	$-0.001$ (n.s.)	+0.019 (n.s.)	+11.9	+4.8

Table 8: Component ablations (Qwen3-4B, C3 budget-matched greedy configuration, 8,192-token budget, vLLM; three repetitions per group, n=200 per task). Official score mean

\pm

std;

\Delta

and hit-rate change vs. G0 (paired Wilcoxon; n.s. =

p>0.05

Model	Configuration	Syn.	Pat.
Qwen3-4B	zero-shot	0.481 $\pm$ 0.005	0.455 $\pm$ 0.010
Qwen3-4B	CoT (greedy)	0.487 $\pm$ 0.004	0.471 $\pm$ 0.005
Qwen3-4B	self-consistency ( $t{=}0.7$ )	0.479	0.405
Qwen3-4B	+ agent (full protocol)	0.365 $\pm$ 0.013	0.335 $\pm$ 0.010
K3	zero-shot	0.496 $\pm$ 0.218	0.556 $\pm$ 0.043
K3	self-consistency ( $t{=}1.0$ )	0.631	0.596

Table 9: Standard reasoning baselines vs. the agent (same-model, official-script score; three repetitions). K3 self-consistency uses the existing temperature-1.0 repetitions; 4B self-consistency samples at temperature 0.7.

$\kappa$	Syndrome	Pathogen.	$\Delta$ syn.	$\Delta$ pat.	$p$ (pat.)
$\tfrac{1}{4}$	0.368	0.333	$-0.012$	$+0.010$	n.s.
$\tfrac{1}{2}$ (deployed)	0.380	0.323	—	—	—
$\tfrac{3}{4}$	0.403	0.361	$+0.024$	$+0.037$	0.036

Table 10: Stopping-fraction ablation (Qwen3-4B, budget-matched greedy, 8,192-token budget, vLLM; n=200 per task, one repetition per

\kappa

). Official scores;

\Delta

and

p

vs. the deployed

\kappa{=}\tfrac{1}{2}

(paired Wilcoxon with bootstrap CI). The

\kappa{=}\tfrac{1}{2}

row is a single run of the G0 configuration and sits within G0’s three-repetition run-to-run spread (

0.365\pm 0.013

syndrome, Table 8 ); the within-table

\Delta

contrasts are unaffected since all three

\kappa

arms share the same single-repetition protocol.

Property	Internalized reasoning	External protocol (ours)
Manipulation resistance	effective [ 26 , 27 ]	ceiling-restoring, $>$ instruction (Sec. 5.6 )
Cross-model transfer	no (bound to training)	yes (model-agnostic adapter)
Auditability	black-box	white-box (cell-level logs)
Failure mode	error entrenchment [ 27 ]	over-prediction (Sec. 5.7 )
Correctability	not evidence-revisable	re-examination loop

Table 11: Two defense mechanisms against manipulation of reasoning. Our protocol targets the right-hand column; the misinformation-benchmark evaluation is reported in Section 5.6 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Certification	task	$n$	zero-shot	agent	$\Delta$ [95% CI]	Wilcoxon $p$
strict	syn.	17	0.183	0.647	$+0.464$ [ $+0.332$ , $+0.597$ ]	3e-4
strict	pat.	7	0.089	0.326	$+0.237$ [ $+0.140$ , $+0.335$ ]	0.036
typical	syn.	136	0.138	0.507	$+0.369$ [ $+0.327$ , $+0.411$ ]	4e-22
typical	pat.	87	0.136	0.535	$+0.399$ [ $+0.350$ , $+0.449$ ]	2e-15

Appendix

Table 12: Budget-certified rescue-prone subsets (DS-Flash, C2; paired per-case differences on the run-mean official score). Strict : agent terminates below the rate-free bound (66.9 s/64.6 s) in every repetition. Typical : agent terminates in every repetition with the median repetition below the bound.

Explore similar work

The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning
May 3, 2026Kwan Soo ShinReasoning Evaluation Evidence-Grounded Reasoning
PolymathMinds AI Lab
RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
Sep 28, 2026Jianpeng Zhao, Haihua Xu, Haoyang Zhang +7LLM Safety Benchmarks LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China
Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems
Aug 4, 2026Sebastián Andrés Cajas Ordóñez, Agastya Munnangi, Aldo Marzullo +13AI Agent Reliability Multi-Agent LLM Systems
MIT Critical Data, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States · Georgia State University, Atlanta, Georgia, United States · Politecnico di Milano, Milan, Italy +12

May 3, 2026J/K move · Enter open · S save

The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning

Kwan Soo Shin

PolymathMinds AI Lab

Reasoning Evaluation Evidence-Grounded Reasoning Multi-Agent Debate LLM Reasoning

Open paper AskPDF arXiv

When copies of the same language model are prompted to debate, they produce diverse phrasings of one perspective rather than diverse perspectives. Multi-agent debate (MAD), and more broadly closed-system reasoning where agents iteratively transform each other's outputs, tends to preserve answer accuracy while degrading the reasoning behind those answers. We name the multi-agent case the Debate Trap and the broader phenomenon the Reasoning Trap, offering a programmatic theory of evidence-grounded reasoning failure.The framework has three parts: (i) SFS (Supported Faithfulness Score), a claim-level metric verifying decomposed atomic claims against provided evidence (decomposer-invariant rankings: Spearman rho=1.0); (ii) EGSR (Evidence-Grounded Socratic Reasoning), replacing adversarial argumentation with evidence-grounded inquiry; (iii) Theorem 1 (DPI Bound): under standard MAD, the chain E -> O^0 -> O^1 -> ... is Markov, and the Data Processing Inequality implies E[I(E;O^{t+1})] <= E[I(E;O^t)]. Three companion results -- open-system recovery (Theorem 2), EGSR accumulation (Lemma 2), and vote-aggregation floor (Proposition 1) -- partition multi-step LLM reasoning by its information-theoretic relationship to E. Across 16 conditions on SciFact (300 claims) and FEVER (1,000 claims), DebateCV (C13) preserves 88% of baseline accuracy while SFS drops 43%; majority-vote MAD (C15) reduces SFS to 1.7% of baseline (p < 10^{-6}, d = -0.96); EGSR recovers 98%. An R6 cohort study (Korean n=10x30 FEVER; English n=3x200 SciFact) finds inter-rater Fleiss kappa <= +0.018 with 0.8-1.4 Likert intra-rater shifts across language and domain -- the human agreement that faithfulness metrics have been calibrated against is not itself stable. We offer one falsifiable conjecture: any closed-system reasoning protocol preserving Theorem 1's Markov structure is, in expectation, subject to the same DPI bound.

Abstract

Explore similar work

The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning

RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems