Partial-Revelation Panel Forecasting: Predictive Information Gain, Allocation, and Coarsening-Consistent Operators

Bin Wen; Dai-Qiao Zhang; Tien-Ping Tan

cs.LGOct 16, 2026

Partial-Revelation Panel Forecasting: Predictive Information Gain, Allocation, and Coarsening-Consistent Operators

Authors: Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan

Organizations: School of Computer Sciences, Universiti Sains Malaysia, Gelugor 11800, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, Gelugor 11800, Penang, Malaysia

Abstract

We study partial-revelation panel forecasting: given $k$ of $n$ annotator votes on the previous dialogue turn, predict the full count distribution of the next turn's panel. This cross-turn predictive information differs from same-item annotation value. Under any nested revelation mechanism, the population decrement satisfies $V(k)-V(k+1)=I(Y;R_{k+1}\mid X_k)$ ; a fitted model also contributes adjacent excess-risk differences. Coarsened-completely-at-random (CCAR) masking supplies the exchangeable random-rater interpretation used in our operators. We define coarsening-consistent panel operators with a closed-form Dirichlet-multinomial realization, learned completion, and rater-count transfer. Against per-level lookup, cold-start count negative log-likelihood differences are -0.422/-0.166/-0.014 nats on IEMOCAP/M3ED/EmoWOZ. A held-out rater experiment measures same-item value: first and second additional votes reduce NLL by 0.583/0.221, 0.578/0.184, and 0.180/0.069 nats, respectively. Cross-turn and same-item gains are not interchangeable, and CCAR masks are controlled interventions rather than observed arrival processes. Consistency is auditable but does not guarantee predictive improvement; per-level marginalization can outperform a shared table. Target-free allocation beats uniform only where row-level signal is predictable: at budget one it recovers 83--93% of retrospective feature class headroom on IEMOCAP/M3ED but does not improve EmoWOZ. Gains are modest and corpus-dependent, with several small or null partial-revelation differences. Code and aggregate result artifacts accompany the preprint.

Figures & tables

Figure 1: Fitted revelation curves (expected count NLL) for ccprJ and per-level lookup on IEMOCAP (session-level leave-one-session-out (LOSO)) and M3ED (official held-out). The shaded bands are 95% cluster-bootstrap confidence intervals (session clusters for IEMOCAP and dialogue clusters for M3ED). Theorem 2 applies to the population curve; deviations of a fitted curve from monotonicity are fitted-risk diagnostics described by Proposition 1 .

Corpus	$k=0$	$k=1$	$k=2$	$k=3$
IEMOCAP	$-0.422$	$-0.075$	$-0.041$	$+0.021$
	$[-0.505,-0.373]$	$[-0.083,-0.067]$	$[-0.049,-0.032]$	$[0.000,0.046]$
M3ED official	$-0.166$	$+0.006$	$+0.007$	$+0.001$
	$[-0.206,-0.125]$	$[-0.005,0.017]$	$[-0.000,0.014]$	$[-0.011,0.012]$
EmoWOZ official	$-0.014$	$-0.006$	$-0.006$	$-0.002$
	$[-0.023,-0.003]$	$[-0.011,-0.001]$	$[-0.010,-0.001]$	$[-0.010,0.006]$

Table 1: Count NLL difference ccprJ

-

lookup (nats), by corpus and revelation level. Negative favors ccprJ. Bold: 95% cluster bootstrap CI excludes zero in ccprJ’s favor. Full NLL values in Appendix B .

Corpus	$0\to 1$	$1\to 2$
IEMOCAP	$0.583$ $[0.565,0.600]$	$0.221$ $[0.197,0.243]$
M3ED	$0.578$ $[0.546,0.612]$	$0.184$ $[0.174,0.195]$
EmoWOZ	$0.180$ $[0.167,0.193]$	$0.069$ $[0.066,0.072]$

Table 2: Same-item held-out rater NLL reductions (nats). Each cell is the reduction from

k

k+1

revealed votes for a held-out annotator; 95% block-bootstrap CIs are in brackets.

Corpus	Control	$0\to 1$	$1\to 2$	$2\to 3$
IEMOCAP	partial majority + relation	$0.545$ $[0.498,0.591]$	$0.116$ $[0.103,0.131]$	$-0.042$ $[-0.077,-0.018]$
	full composition + relation	$0.545$ $[0.498,0.591]$	$0.116$ $[0.103,0.131]$	$0.048$ $[0.009,0.089]$
M3ED	partial majority + relation	$0.640$ $[0.597,0.686]$	$0.272$ $[0.236,0.308]$	$-0.172$ $[-0.202,-0.142]$
	full composition + relation	$0.640$ $[0.597,0.686]$	$0.272$ $[0.236,0.308]$	$0.279$ $[0.249,0.309]$
EmoWOZ	partial majority + relation	$0.057$ $[0.047,0.067]$	$0.025$ $[0.018,0.032]$	$-0.016$ $[-0.024,-0.008]$
	full composition + relation	$0.057$ $[0.047,0.067]$	$0.025$ $[0.018,0.032]$	$0.005$ $[-0.001,0.012]$

Table 3: Inertia controls: adjacent NLL reductions (nats) for majority and full partial-composition controls, with 95% block-bootstrap CIs. Ties in partial majority are retained as ties. The first step is shared because one revealed vote determines the partial majority.

Mechanism	$n$	Train Size	Bias Range	Max RMSE	Violation
CCAR	3	300	$[-0.137,-0.086]$	0.147	0.80
CCAR	3	1000	$[-0.081,-0.036]$	0.083	0.20
CCAR	3	3000	$[-0.030,-0.018]$	0.032	0.00
CCAR	5	1000	$[-0.115,-0.078]$	0.118	1.00
identity slot	3	1000	$[-0.074,-0.042]$	0.076	0.00
label-dependent	5	1000	$[-0.118,-0.087]$	0.118	1.00

Table 4: Synthetic sensitivity at

\rho=0.5

(five replicates). Bias and RMSE are ranges/maxima across adjacent increments; violation is the fraction of replicates whose fitted curve violates monotonicity at any level.

Figure 2: Training data scan: shared consistent table minus per-level inconsistent marginalizer (nats; positive = shared worse; error bars are 95% CIs). ( a ) Overall cost shrinks as data shrinks. ( b ) Sparse cells flip: IEMOCAP

k=2

and EmotionLines

n=5

k=4

favor the shared table as per-level cells become sparser.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	ccprJ	ccprD	Lookup	Flat
0	3.2482	3.2482	3.6703	3.6836
1	2.6903	2.6903	2.7654	3.6341
2	2.5446	2.5446	2.5854	3.6341
3	2.5406	2.5195	2.5195	3.6341

Appendix

Table 5: IEMOCAP count NLL by revelation level (session LOSO).

$k$	ccprJ	Lookup	ccprJ $-$ Lookup
0	2.5266	2.6925	$-0.166$ $[-0.206,-0.125]$
1	1.9721	1.9665	$+0.006$ $[-0.005,0.017]$
2	1.7192	1.7121	$+0.007$ $[-0.000,0.014]$
3	1.6002	1.5997	$+0.001$ $[-0.011,0.012]$

Appendix

Table 6: M3ED official count NLL.

$k$	ccprJ	Lookup	ccprJ $-$ Lookup
0	2.0429	2.0565	$-0.014$ $[-0.023,-0.003]$
1	1.9573	1.9636	$-0.006$ $[-0.011,-0.001]$
2	1.9318	1.9373	$-0.006$ $[-0.010,-0.001]$
3	1.9158	1.9183	$-0.002$ $[-0.010,0.006]$

Appendix

Table 7: EmoWOZ official count NLL.

Corpus	$k=0$	$k=1$	$k=2$	$k=3$
IEMOCAP	$+0.048$	$+0.034$	$+0.006$	$+0.028$
	$[0.044,0.052]$	$[0.029,0.038]$	$[-0.000,0.011]$	$[0.017,0.042]$
M3ED official	$+0.041$	$+0.015$	$+0.003$	$+0.000$
	$[0.029,0.054]$	$[0.009,0.022]$	$[-0.002,0.008]$	$[-0.008,0.008]$
EmoWOZ official	$+0.004$	$-0.001$	$-0.002$	$-0.001$
	$[-0.004,0.011]$	$[-0.004,0.002]$	$[-0.005,0.001]$	$[-0.005,0.004]$

Appendix

Table 8: Consistency ablation: ccprJ-eq

-

per-level inconsistent marginalizer (nats). Positive = shared table worse. Bold: CI excludes zero.

Corpus	Frac	$k=0$	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$	Overall
IEMOCAP	25%	$+0.042$	$+0.025$	$-0.016$	$+0.035$	—	—	$+0.021$
	50%	$+0.046$	$+0.025$	$+0.009$	$+0.030$	—	—	$+0.028$
	100%	$+0.048$	$+0.034$	$+0.006$	$+0.028$	—	—	$+0.029$
M3ED official	25%	$+0.034$	$+0.007$	$-0.003$	$+0.007$	—	—	$+0.011$
	50%	$+0.040$	$+0.009$	$+0.006$	$+0.013$	—	—	$+0.017$
	100%	$+0.041$	$+0.015$	$+0.003$	$+0.000$	—	—	$+0.015$

Appendix

Table 9: Training data scan: shared

-

inconsistent (nats), by corpus, training fraction, and revelation level; overall is the equal-weight mean over

k

. Bold indicates cells whose 95% bootstrap CIs exclude zero (session/fold bootstrap for IEMOCAP and EmotionLines; dialogue bootstrap for M3ED and EmoWOZ).

Cell	$k=0$	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
$n=3$	$+0.009$	$+0.009$	$-0.003$	$0.000$	—	—
$n=5$	$+0.025$	$+0.025$	$-0.000$	$+0.018$	$-0.002$	$+0.012$

Appendix

Table 10: Mixed-

\{3,5\}

deployment: shared table minus per-

(n,k)

inconsistent marginalizer, by cell (nats; positive = shared worse).

Condition	$k=0$	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
$n=3\to n=5$ (zero-shot)	$-0.011$	$+0.001$	$-0.003$	$-0.016$	$-0.039$	$-0.041$
$n=5\to n=5$	$-0.012$	$+0.001$	$-0.010$	$-0.036$	$-0.084$	$-0.087$
$n=5\to n=3$	$-0.008$	$+0.000$	$-0.004$	$+0.006$	—	—
$n=3\to n=3$	$-0.007$	$+0.002$	$-0.004$	$+0.017$	—	—

Appendix

Table 11: CCPR-N minus the in-corpus per-level lookup at the test rater count, on EmotionLines (nats). Bold: fold-bootstrap CI excludes zero.

Corpus	$B=0.5$	$B=1.0$	$B=1.5$	$B=2.0$	$B=2.5$
IEMOCAP	$+0.168$	$+0.341$	$+0.297$	$+0.239$	$+0.109$
M3ED official	$+0.133$	$+0.248$	$+0.208$	$+0.189$	$+0.088$
EmoWOZ official	$+0.015$	$+0.042$	$+0.041$	$+0.030$	$+0.031$

Appendix

Table 12: Annotation budget: saving of uniform over concentrated allocation (nats/step; concentrated

-

uniform). Bold: CI excludes zero.

Corpus	$B$	CCPR-P	Uniform	Retrospective	Oracle	P $-$ Uniform
IEMOCAP	0.5	2.4132	2.4812	2.3865	2.2462	$-0.068$ $[-0.093,-0.049]$
	1.0	2.3511	2.4449	2.3317	2.1774	$-0.094$ $[-0.135,-0.057]$
	1.5	2.3381	2.4166	2.3207	2.1612	$-0.079$ $[-0.111,-0.057]$
	2.0	2.3500	2.3850	2.3202	2.1786	$-0.035$ $[-0.053,-0.009]$
	2.5	2.3487	2.3630	2.3337	2.2303	$-0.014$ $[-0.023,-0.005]$
M3ED	0.5	2.0357	2.1280	2.0066	1.7877	$-0.092$ $[-0.150,-0.024]$

Appendix

Table 13: CCPR-P: target-free learned water-filling policy versus uniform, per-row mean count NLL. “Retrospective” is the in-sample feature class reference and is not deployable; oracle greedy uses the evaluation rows’ realized curves. The final column gives CCPR-P minus uniform with block-bootstrap CIs (positive = CCPR-P worse). Final-column entries are bold when their 95% block-bootstrap CIs exclude zero.

Corpus	Policy	$B=1.0$	$B=2.0$
IEMOCAP	predictive entropy	$0.000$ $[0.000,0.000]$	$-0.0015$ $[-.0021,-.0007]$
	least confidence	$-0.0037$ $[-.0091,.0013]$	$+0.0011$ $[-.0062,.0067]$
	inverse margin	$-0.0019$ $[-.0055,.0029]$	$+0.0167$ $[.0036,.0312]$
M3ED	predictive entropy	$0.000$ $[0.000,0.000]$	$+0.0009$ $[-.0019,.0040]$
	least confidence	$+0.0125$ $[.0074,.0210]$	$+0.0402$ $[.0286,.0579]$
	inverse margin	$+0.0702$ $[.0288,.1212]$	$+0.0705$ $[.0377,.1001]$

Appendix

Table 14: Mean-label uncertainty allocation controls. Entries are policy minus uniform with 95% block-bootstrap CIs (nats; negative is better).

Corpus	Level	ccprS	Shared	Per-Level	Lookup	S $-$ Inc	S $-$ Shared
IEMOCAP	$k=0$	3.1765	3.2244	3.1765	3.6703	$+0.000$	$-0.048$
	$k=1$	2.6395	2.6729	2.6390	2.7654	$+0.001$	$-0.034$
	$k=2$	2.5256	2.5325	2.5265	2.5854	$-0.001$	$-0.007$
	$k=3$	2.5228	2.5508	2.5226	2.5195	$+0.000$	$-0.028$
M3ED official	$k=0$	2.4740	2.5154	2.4740	2.6925	$+0.000$	$-0.041$
	$k=1$	1.9487	1.9641	1.9487	1.9665	$+0.000$	$-0.015$

Appendix

Table 15: CCPR-S per-level NLL and deltas (nats; positive = ccprS worse; bold = 95% CI excludes zero). Shared = ccprJ-eq for

n=3

, ccprN for

n=5

Corpus	Term	$k=0$	$k=1$	$k=2$	$k=3$
IEMOCAP	total	$-0.0653$	$-0.0618$	$-0.0339$	$+0.0054$
		$[-0.1014,-0.0375]$	$[-0.0892,-0.0400]$	$[-0.0479,-0.0218]$	$[+0.0024,+0.0083]$
	context	$-0.0610$	$-0.0401$	$-0.0147$	$0.0000$
		$[-0.0969,-0.0333]$	$[-0.0616,-0.0219]$	$[-0.0271,-0.0037]$	$[0,0]$
	table	$-0.0043$	$-0.0218$	$-0.0192$	$+0.0054$
		$[-0.0072,-0.0013]$	$[-0.0272,-0.0170]$	$[-0.0226,-0.0154]$	$[+0.0024,+0.0083]$

Appendix

Table 16: CCPR-L decomposition, mean [95% CI].

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 135 papers

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
Sep 29, 2026Jingjie Ning, Xueqi Li, Yibo Kong +1Forecasting Benchmarks Probabilistic Forecasting
Carnegie Mellon University · Tsinghua University
Available Guardrails: Certifying Selective Prediction across ML Systems
Sep 18, 2026Parivesh Priye, Yufeng Wang, Haibin Ling +1Selective Prediction Language Model Safety Evaluation
Rivian and Volkswagen Group Technologies · Stony Brook University · Westlake University
Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Aug 7, 2026Yang ShuLLM Evaluation LLM-as-a-Judge
Zhejiang University, Hangzhou, China

Sep 29, 2026J/K move · Enter open · S save

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li

Carnegie Mellon University · Tsinghua University

Forecasting Benchmarks Probabilistic Forecasting AI Agents for Scientific Discovery

Open paper AskPDF arXiv

Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet (

D-M=-.0026

, 95 percent interval [

-.0174

, .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.

Abstract

Human-in-the-Loop Annotation

Explore similar work

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Available Guardrails: Certifying Selective Prediction across ML Systems

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps