Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Zhelun Wu

cs.AIAug 14, 2026

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Authors: Zhelun Wu

Organizations: Atlassian

Abstract

Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.

Figures & tables

Panel of Figure 1 — Figure 1: The partition. Each source is read alone under a causal context constraint and must emit the same four-field evidence tuple; a fixed rule then combines tuples into a decision that includes abstention. The dashed line is the design surface: everything left of it can be replaced without touching the arithmetic, and everything right of it can be corrected without re-reading a source.

Panel of Figure 2 — Figure 2: Count-scale drift ( Proposition 2 ), plotted in posterior log-odds. A vote threshold is a posterior threshold, but not a fixed one: the bar it enforces falls linearly in the number of sources consulted, at rate $\log\frac{1-\alpha}{1-\beta}$ . More reliable readers drift faster — at $(\alpha,\beta)=(0.95,0.05)$ the effective bar drops by almost three log-odds per source. Curves are the exact values of Equation 3 at $\tau^{\prime}=1$ and prior $0.2$ ; nothing is fitted.

Result	Needs	Fails when
Proposition 1 (bonus is redundant)	Uniform weights; $h$ increasing; $\lambda\geq 0$	Weights heterogeneous — then the bonus does change decisions, though with no principle fixing the exchange rate
Proposition 2 (count-scale drift)	Informed silence; homogeneous rates; conditional independence	Positive-only firing ( $D=0$ ); or heterogeneous buckets, where Proposition 3 governs
Proposition 3 (misweighting)	Informed silence; conditional independence	Weights already proportional to $g_{b}$ ; or one bucket only
Corollary 1 (scope)	Every channel consulted per instance	Channels omitted rather than silent, as in retrieval
Corollary 2 (precision declines in $n$ )	The above, plus prevalence not varying with $n$	Prevalence trends with $n$ , which must be measured, not assumed
Proposition 4 (exactness)	Conditional independence; exact rates; $\kappa=1$ ; ancillary buckets	Any of the four; bucket ancillarity is the one we expect to fail ( Remark 3 )

Table 1: Assumptions and failure conditions for the results of this section. Conditional independence given the label is assumed by every result except Proposition 1 , and is the premise our own causal-context design violates by construction — which is what the dependence discount is for.

Method	Precision	Recall	Instance $F_{1}$	Macro $F_{1}$
Prior only	0.697	0.199	0.303	0.114
One-shot	0.645	0.607	0.614	0.425
Independent, no context	0.806	0.418	0.520	0.385
Independent, full context †	0.816	0.439	0.534	0.392
Independent, causal context	0.824	0.459	0.587	0.416
Causal $+$ prior fallback	0.772	0.554	0.614	0.466

Table 2: Reading strategies, aggregation held fixed.

n=33

evaluation instances ( Section 6.2 ); bold marks the numerically best value in a column and implies no significance, since instance-level intervals at this

n

overlap for every pair. Instance

F_{1}

is a tie between one-shot and the recommended configuration, and both are marked. † Oracle: uses future sources, not deployable, and not length-matched to the causal condition (see Section 6.5 ).

#	Configuration	Feat.	AUPRC ${}_{\text{s}}$	AUPRC ${}_{\text{f}}$	$F_{1}$	MAE (d)	$\Delta$
0	Random (prevalence) ‡	0	0.185	0.171	0.270	80.9	$-$ 0.620
1	Cadence only	5	0.505	0.327	0.428	53.3	$-$ 0.300
2	Hand-crafted (baseline)	18 ¶	0.805	0.667	0.700	50.6	—
3	$+$ conversation quality	25	0.805	0.675	0.702	50.6	$+$ 0.000
4	$+$ all structured engagement	34	0.804	0.671	0.704	50.6	$-$ 0.001
5	GRU $+$ cadence	69	0.833	0.679	0.725	51.8	$+$ 0.028

Table 3: Identical training data and held-out temporal split.

\Delta

is change in AUPRC

{}_{\text{s}}

vs. row 2. ‡ Prevalence on the resolved population used for ranking metrics, not the unconditional 90-day prevalence of Section 7 ; this is the correct no-skill reference for every row here. § Row 7 is not feature-matched to row 6 and therefore does not isolate architecture, and no feature-matched neural arm was run. ¶ Feature count unreconciled: the enumerated set is 11 numeric

+

6 categorical

=17

, and row 6’s 81

=64+17

agrees, but this row is logged as 18 and rows 3–4 as 25 and 34; the offset propagates and must be fixed. As in Table 2 , bold marks the numerically best value in a column and implies no significance: no column reports instance-level intervals. MAE is descriptive and is not bolded, since we do not use it to rank configurations.

Sources observed	Instances	AUPRC ${}_{\text{s}}$	AUPRC ${}_{\text{f}}$	$F_{1}$	MAE (d)
1–2	2,830	0.795	0.622	0.699	59.4
3–4	1,043	0.802	0.680	0.696	51.9
5–7	571	0.827	0.697	0.718	42.5
8–12	304	0.862	0.788	0.765	38.5
13 $+$	126	0.940	0.883	0.849	39.0

Table 4: Accuracy as sequential evidence accumulates. The monotone trend is consistent with the sequence-encoder rationale, but it is not identified: instances with more sources are also further along, so proximity to resolution is an unmodelled competing explanation ( Section 7.7 ). Instance counts in the upper bins are small and intervals are pending. The 1–2 bin count (2,830) coincides exactly with the population used for the operating-point table, which we flag as a probable provenance error to be checked rather than a coincidence.

Domain	Sources	Reliability bucket from	Dependence blocks	Regime
Clinical decision support	Notes, labs, imaging reports, prior visits	Modality and assay precision; report completeness	Same encounter; same instrument; copied-forward text	Both
Security triage	Detector alerts, host telemetry, threat intel	Detector historical FPR; alert enrichment depth	Shared sensor, shared rule lineage	Unresolved
Financial analysis	Filings, transcripts, analyst notes	Document class and recency; quantitative grounding	Same issuer-period; syndicated language	Both
Evidence synthesis	Published studies of varying quality	Design and risk-of-bias tier; sample size	Same cohort; same research group	Resolved
Industrial telemetry	Per-signal detectors over a shared feature build	Signal type; data-quality flag	Shared upstream pipeline or reference series	Unresolved

Table 5: Instantiating the contract along the four axes of Section 3 . Each domain aggregates multiple partially reliable signals under an obligation to justify the conclusion, and in several of them the conventional method is to sum scores against a fixed cut — the form Corollary 1 shows to be a drifting posterior threshold.

Explore similar work

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions
Sep 30, 2026Haoyang Zhang, Jianpeng Zhao, Qi Hao +1LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China
Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?
Oct 3, 2026Nishanth Nayakanti, Prasang Gupta, Ashutosh Bilthare +1Language Model Post-Training Rejection Sampling
PricewaterhouseCoopers, U.S.
Evidence Integration in Large Language Models
Sep 3, 2026Sebastien Kawada, Manolis KellisBelief Updating in LLMs Knowledge Augmentation for Language Models
Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

Sep 30, 2026J/K move · Enter open · S save

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions

Haoyang Zhang, Jianpeng Zhao, Qi Hao, Pengyang Wang

State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China

LLM Decision-Making

Open paper AskPDF arXiv

Rule-based reasoning, as in eligibility checks and contract reviews, requires language models to assess evidence against individual conditions and combine their judgments under explicit rules. Errors in evidence assessment can leave a decision unchanged, but misinterpreting or overlooking decision-critical evidence can reverse it. Identifying such evidence allows more capable models to focus on checking the corresponding condition judgments, supporting accurate and safe decisions. Recognizing the evidence's criticality requires understanding how evidence affects a condition judgment and how that judgment affects the decision. To achieve the goal, we propose a INTERvention-based imPACT learning framework (InterPact), which enables counterfactual verification of evidence criticality in rule-governed decisions. Specifically, its evidence intervention constructor generates training pairs for a propagation verifier by editing case facts with a frozen language model while holding rules and non-target conditions fixed. Human-reviewed labels record the resulting condition and decision changes, while complete state-to-decision mappings supervise consequences beyond the observed edit. During training, the verifier weights learned conditional decision predictions by evidence-based condition probabilities through a fixed composition operation, propagating decision-change supervision into the base model. At inference, the trained base model directly judges criticality from the original case and target evidence, without human or stronger-model supervision. On single-case evidence criticality verification over adapted rule-governed decision cases, InterPact achieves 68.28% accuracy, outperforming all six baselines. These results support learned decision sensitivity as a basis for prioritizing evidence checks.

Abstract

Explore similar work

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions

Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?

Evidence Integration in Large Language Models