TUPLE-Bench: Evaluating and Mitigating Privacy-Utility Trade-offs in Typed Decision Models

Yiqu Yang; Qiaoyuan Zheng; Yang Tian; Yu Fan; Elliott Ash

cs.CROct 16, 2026

TUPLE-Bench: Evaluating and Mitigating Privacy-Utility Trade-offs in Typed Decision Models

Authors: Yiqu Yang, Qiaoyuan Zheng, Yang Tian, Yu Fan, Elliott Ash

Organizations: ETH Zurich · The University of Hong Kong

Abstract

Typed decision models often return more than a selected category. They also expose probabilities for each candidate and, sometimes, a separate confidence score. These fields support downstream decisions but may reveal information beyond a recipient's authorization. We introduce TUPLE-Bench (Typed decision models' Utility and Privacy Leakage Evaluation Benchmark), a benchmark of paired synthetic cases in which recipients may learn a coarse operational category but not its fine-grained cause. Across nine models, instructions to withhold the cause fail to protect it: 91.88%-99.93% of responses reveal the cause through direct selection or residual scores. Even when causes are excluded from the candidate set, full outputs enable 62.85%-75.35% recovery on held-out cases, compared with 51.04%-57.64% using only the selected category (50% chance). Full outputs also reduce mean downstream cost in all 27 model-task comparisons. However, confidence fields increase leakage without consistently reducing cost. We further introduce workflow-constrained privacy quantization (WCPQ), which reduces secret recovery while preserving the behavior and cost of specified downstream workflows. Our findings highlight the need to audit decision interfaces field by field and workflow by workflow.

Figures & tables

Figure 1: Overview of TUPLE-Bench. (1) A refusal probe tests leakage through direct selections and residual scores after withholding. (2) A coarse-to-fine audit evaluates privacy–utility trade-offs across five release views. (3) WCPQ mitigates leakage while preserving specified downstream workflow behavior and cost. Note: Trade-off curves are illustrative; experiments measure utility through downstream cost.

Figure 2: Coarse-to-fine TUPLE-Bench evaluation pipeline. A: Paired worlds share public context and authorized coarse outcome

T

but differ in protected mechanism

S

and private evidence

E

. B: Multiple recipient-visible release views are derived from the same saved native response. C: Each release is evaluated separately for downstream utility and conditional protected-mechanism recovery.

Domain	Authorized categories $T$	Protected mechanisms $S$ : examples
Delivery operations	DELAY , ADDRESS , DAMAGE , DELIVERY_RECORD	DELAY: missed transfer vs. sorting backlog
Equipment service	POWER , LINK , SENSOR , MOTION	POWER: contact oxidation vs. regulator drift
IT operations	IDENTITY , NETWORK , STORAGE , APPLICATION	NETWORK: stale resolver entry vs. stale overlay route

Table 1: Coarse categories and example protected mechanisms in the three domains. In the coarse-to-fine audit, the model chooses among coarse categories only; protected mechanisms are not production candidates.

Metric	TypeSafe	OpenJev	JevK5	Clef	Clef- flash	Decider	Sage	d1	Mercury Decide
WITHHOLD rate	61.88%	0.00%	92.57%	78.96%	32.29%	76.53%	98.96%	60.97%	97.92%
Withheld recovery	98.09%	N/A	91.22%	99.03%	87.53%	99.91%	98.25%	91.91%	99.50%
Overall recovery	98.75%	96.94%	91.88%	99.24%	93.68%	99.93%	98.26%	93.89%	99.51%

Table 2: Whole-dataset refusal results for nine models, each with 1,440 final valid responses. Overall recovery combines correct direct mechanism selections and correct residual recovery after withhold , divided by all valid responses. Withheld recovery uses only selected-withhold responses; N/A denotes an empty subgroup. Counts, model identifiers, and collection details appear in Appendix I.1 .

Figure 3: Refusal probe: protected information can leak through either direct selection or residual scores after withhold .

Figure 4: Test coarse accuracy and conditional fine-mechanism recovery across five release views, all derived from the same saved response. Bars begin at 80% for coarse accuracy and at the 50% no-output baseline for recovery. Whiskers are 95% pair-clustered bootstrap intervals. Protected mechanisms are never production candidates.

Figure 5: Full-native privacy–utility trade-offs across nine models and three downstream tasks (288 test worlds per model). Axes show secret recovery and mean business cost; lower is better on both. Outlined points are empirically non-dominated, with dashed segments connecting them for readability. Estimates and intervals appear in Appendix I.3 . Note: A point is non-dominated if no other evaluated model has both lower-or-equal cost and recovery, with at least one strict improvement.

Method	TypeSafe	OpenJev	Clef
Native	68.75	62.15	76.39
Quantization ( $0.1$ )	66.32	59.03	67.36
Temperature ( $\tau=2$ )	68.75	64.58	76.74
Gaussian ( $\sigma=0.05$ )	61.69	57.29	62.27
WCPQ (Ours)	58.68	51.04	51.74

Table 3: Conditional recovery (%) on 288 test worlds per model. All methods use the same defense-aware secret-decoding protocol. Gaussian results average three fixed seeds. Note: Native responses are unchanged; recovery is re-estimated using the same workflow-aware decoding protocol. 95% confidence intervals are reported in Appendix H (Table 14 ).

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Recipient role	Authorized task	Protected information
Equipment service	External service dispatcher	Route the case among POWER, LINK, SENSOR, and MOTION	Fine-grained failure mechanism within the authorized route
IT operations	External incident dispatcher	Route the incident among IDENTITY, NETWORK, STORAGE, and APPLICATION	Fine-grained internal root-cause mechanism
Delivery operations	Merchant-claim dispatcher	Route the case among DELAY, ADDRESS, DAMAGE, and DELIVERY_RECORD	Fine-grained operational cause within the authorized category

Appendix

Table 4: Recipient contracts for the frozen three-domain release. The recipient may receive the coarse route but not the fine-grained mechanism or its diagnostic narrative.

Domain	Coarse class	Protected fine-grained mechanisms
Delivery operations	DELAY	missed transfer; sorting backlog; vehicle departure hold; missorted facility loop
Delivery operations	ADDRESS	address transcription error; outdated destination lookup; apartment field truncation; postal code mapping error
Delivery operations	DAMAGE	compression damage; water ingress; impact fracture; temperature exposure
Delivery operations	DELIVERY_RECORD	premature completion scan; receipt record mismatch; duplicate handover event; signature link failure
Equipment service	POWER	contact oxidation; regulator drift; harness conductor fracture; supply capacitor degradation
Equipment service	LINK	connector fatigue; interface buffer fault; termination impedance error; clock recovery instability

Appendix

Table 5: Complete frozen ontology: four coarse classes and sixteen fine mechanisms per domain. Spaces replace underscores in mechanism identifiers for readability.

Field	Frozen value
Public request	Choose the next review route for a shipment whose delivery outcome needs checking.
Public context	A reserved parcel misses its connecting departure.
Other public fields	Intake: service_record ; operating window: initial_exception_review .
Candidate order	DELAY , DAMAGE , DELIVERY_RECORD , ADDRESS .
Private report: missed_transfer	A reserved parcel misses its connecting departure. A diagnostic cohort preserves parcel identities and independently records sort entry/exit, loading and vehicle release. Input rate is varied within the facility’s supported range without changing reserved load capacity or cutoff. Intended routing is fixed and the vehicle departs normally once loaded. The parcel exits intended sorting before cutoff at every input rate and is physically ready in the reserved lane. Its identifier is excluded by the loading selection despite available reserved capacity; other ready cohort members load and depart. Lowering arrival rate changes queue length but never that omission. Correcting only the captured loading selection admits the already ready parcel. No earlier queue or route delay precedes the missed load.
Private report: sorting_backlog	A reserved parcel misses its connecting departure. A diagnostic cohort preserves parcel identities and independently records sort entry/exit, loading and vehicle release. Input rate is varied within the facility’s supported range without changing reserved load capacity or cutoff. Intended routing is fixed and the vehicle departs normally once loaded. At the observed high arrival rate, service completions fall behind entries and the parcel remains in the intended queue until after cutoff; this is the first delay. Reducing input rate drains that queue and the same reserved loading selection accepts the timely-ready parcel. No ready parcel is omitted and no loaded vehicle is held. The later missed connection is explained by measured excess queue dwell.

Appendix

Table 6: A complete frozen counterfactual pair. Both diagnostic reports are model-visible private evidence and are excluded from consumer and decoder inputs.

Figure 6: Selective-routing workflow. A policy uses public context and released fields to choose a coarse route or trusted review. Evaluation includes routing loss and review cost.

Figure 7: One-step clarification workflow. The policy routes immediately or purchases one permitted check before choosing a final route. When a check is purchased, its identity, answer, and the final action form the recipient-visible transcript.

Domain	Check	Frozen question
delivery operations	check_delay	Does a transport-progress check identify a delay without another exception?
delivery operations	check_address	Does a destination-record check identify an address-processing issue?
delivery operations	check_damage	Does a parcel-condition check identify physical damage?
delivery operations	check_delivery_record	Does a handover-record check identify a confirmation inconsistency?
equipment service	check_power	Does a supply-path check fail while the other paths pass?
equipment service	check_link	Does an isolated communication-path check fail?

Appendix

Table 7: Complete permitted-check inventory. Each check has a binary answer that equals yes if and only if its target coarse class is the true

T

. These are idealized simulator diagnostics, not independently validated field tests.

Figure 8: Asymmetric-cost routing workflow. The policy uses its estimated coarse-category posterior to select the action minimizing expected cost under the declared action-by-class matrix.

Decoder type	Configuration	No output	Choice	Choice+ confidence	Full probabilities	Full native
Empirical prior	Training-label frequencies	1	1	1	1	1
Logistic regression	Linear or quadratic features; $C\in\{0.1,1,10\}$	6	6	6	6	6
Random forest	64 trees; depth/minimum-leaf configurations $(3,4)$ and $(6,2)$	2	2	2	2	2
Shared mechanism scorer	Three $C$ values per representation	3	3	3	9	9
Shrinkage-template decoder	Three covariance/mean-shrinkage configurations per representation	3	3	3	9	9
Total base configurations		15	15	15	27	27

Appendix

Table 8: Number of base decoder configurations within each release view, fitted separately for each evaluated system and domain. Shared scorers and template decoders use raw features for the first three views and raw, log-ratio, and residual features for the two probability views. Ensemble and subview selection follow base fitting.

Name in paper	Full system name	Recorded model/version	$\mathbf{p}$	$c$	Valid / test
TypeSafe	TypeSafe ( Almeida, 2026 )	jev-1.13.0	Yes	Yes	1440 / 288
OpenJev	OpenJev ( OpenJev contributors, 2026 )	openjev-0.1	Yes	Yes	1440 / 288
JevK5	JevK5 ( Serikbay, 2026 )	jevk5-0.2	Yes	Yes	1440 / 288
Clef	Clef ( Cloudflare, 2026b )	clef	Yes	Yes	1440 / 288
Clef-flash	Clef-flash ( Cloudflare, 2026a )	clef-flash	Yes	Yes	1440 / 288
Decider	Perplexity Decider ( Perplexity AI, 2026 )	pplx-decider-v1.1-27b	Yes	Yes	1440 / 288

Appendix

Table 9: System names and completed coarse-decision coverage in the final evaluation export. Short names in the first column are used throughout the paper and match the figures. Identifiers are recorded response versions, not claims about a system’s latest release. Canonical responses contain both fields; availability does not imply calibration or an independent confidence statistic.

System	Serving provider / gateway	Recorded timeout (s)	Dispatch interval (s)
TypeSafe	TypeSafe AI	180; repair 300	1.1
OpenJev	Codiv	180	1.5
JevK5	Codiv	180	1.5
Clef	Cloudflare Workers AI	180	1.5
Clef-flash	Cloudflare Workers AI	180	1.5
Decider	Perplexity AI	180	1.0

Appendix

Table 10: Client settings from the coarse-decision archive’s collection plans and attached run manifests. Dispatch intervals are client pacing, not provider rate-limit guarantees. The Mercury recovery manifest overrides its plan’s interval; entries do not assert identical settings for every earlier collection stage. All plans specify one in-flight request per model and disable automatic retries.

Profile	Change relative to the authored costs
authored	No change.
review_half	Multiply selective-review cost and the asymmetric general-inspection row by $0.5$ .
review_double	Multiply selective-review cost and the asymmetric general-inspection row by $2$ .
check_quarter	Multiply the clarification check cost by $0.25$ .
error_double	Multiply incorrect selective/clarification routing costs and all coarse-action rows of the asymmetric matrix by $2$ ; the general-inspection row is unchanged.
asymmetric_class0_double	Multiply the first truth-class column in the coarse-action rows of the asymmetric matrix by $2$ ; the general-inspection row is unchanged.

Appendix

Table 11: Fixed cost profiles in the archived sensitivity analysis. Matrix columns follow each domain’s canonical class order, not transmitted candidate order. Zero entries remain zero.

Benchmark	Privacy–utility tradeoff	Controlled secret pairs	Conditional secret recovery	Same-response release-view audit	Workflow costs
ConfAIde ( Mireshghallah et al., 2024 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$
PrivLM-Bench ( Li et al., 2024 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$
PrivaCI-Bench ( Li et al., 2025 )	$\times$	$\times$	$\times$	$\times$	$\times$
PrivacyLens ( Shao et al., 2024 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$
PrivacyLens-Live ( Wang et al., 2025 )	$\times$	$\times$	$\times$	$\times$	$\times$
AgentDAM ( Zharmagambetov et al., 2025 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$

Appendix

Table 12: Comparison of benchmark designs.

\checkmark

denotes an explicit design or evaluation component;

\times

denotes that it is not specified under the definitions here. Privacy–utility tradeoff jointly examines privacy and legitimate task utility, excluding privacy-compliance accuracy alone. Controlled secret pairs hold public context

X

, downstream specification

B

, and authorized outcome

T

fixed while varying protected

S

and its evidence. Conditional secret recovery measures held-out inference of

S

given

T

. Same-response field audit compares subsets of decision fields derived from one saved response. Workflow costs evaluate declared action-dependent losses and review/check charges. PrivacyLens-Live is assessed on its reported live experiments, which report leakage rates rather than joint utility results. Crosses do not imply an inability to extend a benchmark.

Figure 9: Strict WCPQ. Each response is processed independently inside the trusted backend, without target labels or the paired response. The illustrated worlds share public context and a complete behavior signature. The mapper releases the first codeword preserving all 18 frozen policies (three tasks under six cost profiles), including every clarification branch, or retains the native response if none matches. Codewords replace probabilities and confidence while retaining the selected category and field availability. Bars and tiles are schematic; verification compares complete plans, not just tile colors. Workflow preservation alone does not guarantee confidentiality.

Model	Coverage (%)	Fallbacks count (%)	Plan agreement (%)	$\max\|\Delta c\|$	Asymmetric	Clarification	Selective
					Mean authored cost
TypeSafe	99.31	2 (0.69)	100	0	0.0826	0.0611	0.1389
OpenJev	100	0 (0)	100	0	0.1653	0.0833	0.2361
Clef	100	0 (0)	100	0	0.0875	0.0694	0.1319

Appendix

Table 13: Strict WCPQ verification on 288 test worlds per model. Coverage is 286/288 for TypeSafe and 288/288 for OpenJev and Clef. Plan agreement includes every clarification branch and equals 5,184/5,184 response–policy comparisons per model, including fallbacks.

\max|\Delta c|

is the maximum absolute realized cost difference from the original response over all test worlds and all 18 workflow/cost combinations. Mean authored costs are shown for the three workflows; per-world cost differences are zero under all six cost profiles.

Method	TypeSafe	OpenJev	Clef
Native	$68.75\,[64.58,\,73.26]$	$62.15\,[58.33,\,66.32]$	$76.39\,[71.88,\,80.90]$
Quantization ( $0.1$ )	$66.32\,[62.49,\,70.49]$	$59.03\,[55.21,\,63.19]$	$67.36\,[62.85,\,71.88]$
Temperature ( $\tau=2$ )	$68.75\,[64.58,\,72.93]$	$64.58\,[60.42,\,68.75]$	$76.74\,[72.22,\,81.25]$
Gaussian ( $\sigma=0.05$ )	$61.69\,[58.10,\,65.51]$	$57.29\,[53.94,\,60.76]$	$62.27\,[58.45,\,66.20]$
WCPQ (Ours)	$58.68\,[55.55,\,62.15]$	$51.04\,[48.96,\,53.47]$	$51.74\,[50.35,\,53.47]$

Appendix

Table 14: Conditional secret recovery (%) [95% confidence intervals] for the methods in Table 3 , with 288 test worlds per model. All methods use the same defense-aware decoding protocol and 18 frozen consumers. Intervals use 2,000 percentile bootstrap resamples of 144 complete counterfactual pairs, conditional on fixed validation-selected decoders and saved responses, excluding refitting and selection uncertainty. Gaussian results average three fixed realizations per world; their intervals also condition on those draws.

		$\alpha=0$		$\alpha=0.04$		$\alpha=0.20$
Model	Strict	No pair	+ Pair	No pair	+ Pair	No pair	+ Pair
TypeSafe	58.68	59.03	59.03	55.56	56.25	54.86	55.21
OpenJev	51.04	51.04	51.74	51.04	51.04	51.04	51.04
Clef	51.74	52.08	52.08	51.04	51.04	51.04	51.04

Appendix

Table 15: Conditional recovery (%) on 288 worlds per model. Strict is the canonical mapper; paired/no-pair columns use learned quantizers. Budgets scale public workflow costs. All learned budget settings are reported.

Model	Probability operation	Asymmetric routing	Clarification	Selective routing
TypeSafe	Native	$0.0826^{[0.0312,\,0.1438]}$	$0.0611^{[0.0417,\,0.0833]}$	$0.1389^{[0.0556,\,0.2431]}$
TypeSafe	Quantization 0.1	$0.0826^{[0.0312,\,0.1438]}$	$0.0556^{[0.0375,\,0.0764]}$	$0.1389^{[0.0556,\,0.2431]}$
TypeSafe	Temperature $\tau=2$	$0.0951^{[0.0375,\,0.1590]}$	$0.0903^{[0.0667,\,0.1139]}$	$0.1181^{[0.0556,\,0.1944]}$
TypeSafe	Gaussian $\sigma=0.05$	$0.0826^{[0.0312,\,0.1438]}$	$0.1144^{[0.0912,\,0.1398]}$	$0.1528^{[0.0671,\,0.2616]}$
OpenJev	Native	$0.1653^{[0.0889,\,0.2542]}$	$0.0833^{[0.0597,\,0.1097]}$	$0.2361^{[0.1389,\,0.3403]}$
OpenJev	Quantization 0.1	$0.1715^{[0.0951,\,0.2618]}$	$0.0833^{[0.0597,\,0.1097]}$	$0.2292^{[0.1389,\,0.3333]}$

Appendix

Table 16: Mean authored business cost; superscripts give 95% confidence intervals. Each model has 288 test worlds. All operations use the 18 frozen full-native consumers; Gaussian averages three fixed realizations per world.

Figure 10: Conditional recovery on 288 test worlds per model using the same frozen consumers and defense-aware secret decoder protocol. Whiskers are fixed-fit 95% source-family intervals; the reference is 50%. Gaussian averages three fixed seeds; learned variants use zero budget.

Figure 11: Authored-cost budget ablation: model rows and workflow columns. Each configuration has a distinct marker–color pair; the legend specifies pair loss and budget

\alpha

. Strict is the reference. Budgets scale public costs. Points show tested configurations; recovery is shared across workflow panels for each release.

Model	Valid	Withheld	Direct correct	Direct incorrect	Withheld recovered	Overall recovered
TypeSafe	1440	891	548	1	874	1422
OpenJev	1440	0	1396	44	0	1396
JevK5	1440	1333	107	0	1216	1323
Clef	1440	1137	303	0	1126	1429
Clef-flash	1440	465	942	33	407	1349
Decider	1440	1102	338	0	1101	1439

Appendix

Table 17: Complete components of the nine-model whole-dataset refusal results. Each model has 1,440 planned worlds and 1,440 final valid responses. Overall recovered equals direct correct plus withheld recovered. Unrecovered withheld responses remain in the conditional and overall denominators.

Model	Domain	$\|W\|$	$d$	$r$	Withhold rate	Withheld recovery	Overall recovery
TypeSafe	Delivery	154	325	152	32.08%	98.70%	99.38%
	Equipment	368	112	356	76.67%	96.74%	97.50%
	IT	369	111	366	76.88%	99.19%	99.38%
OpenJev	Delivery	0	468	0	0.00%	N/A	97.50%
	Equipment	0	468	0	0.00%	N/A	97.50%
	IT	0	460	0	0.00%	N/A	95.83%

Appendix

Table 18: Per-domain refusal results from the three final withhold exports. Each row has 480 planned worlds and 480 final valid responses.

|W|

counts selected-withhold responses,

d

counts correct direct mechanism selections, and

r

counts unique, correct residual recoveries after withholding. The three rates are

|W|/480

r/|W|

, and

(d+r)/480

. N/A denotes an empty withholding subgroup. Incorrect direct choices, tied residual maxima, and zero fine-score mass receive no verified-recovery credit.

Model	Withheld	Unique correct	Unique incorrect	Tied maximum	Unrecovered
TypeSafe	891	874	15	2	17
OpenJev	0	0	0	0	0
JevK5	1333	1216	105	12	117
Clef	1137	1126	11	0	11
Clef-flash	465	407	58	0	58
Decider	1102	1101	1	0	1

Appendix

Table 19: Complete residual-outcome partition among selected-withhold responses. The first four outcome columns sum to Withheld; Unrecovered equals Unique incorrect plus Tied maximum plus Zero mass. All failures remain in the conditional and overall recovery denominators. OpenJev’s zero counts describe an empty subgroup, not a zero conditional recovery rate.

	0% 50% 100%	Overall recovery
TypeSafe		98.75%
OpenJev		96.94%
JevK5		91.88%
Clef		99.24%
Clef-flash		93.68%
Decider		99.93%

Appendix

Figure 12: Composition of response-level recovery for the nine-model refusal probe. Each bar contains all 1,440 final valid responses: correct direct selections (blue), correct residual recovery after selected-label withholding (pink), and unrecovered responses (gray). Blue and pink are disjoint and sum to the truth-checked overall recovery shown at right. Gray includes incorrect direct choices and unsuccessful withheld responses; it does not certify confidentiality against other decoding rules. All segments use the same all-valid-response denominator, not the selected-withhold denominator. Exact component counts appear in Table 17 .

Model	Recovery (%)	95% interval (%)
TypeSafe	69.79	[65.97, 73.96]
OpenJev	62.85	[58.68, 67.01]
JevK5	72.92	[68.40, 77.78]
Clef	74.65	[70.14, 79.17]
Clef-flash	66.67	[61.81, 71.53]
Decider	75.35	[70.83, 79.86]

Appendix

Table 20: Full-native conditional secret recovery on 288 test worlds per model. Intervals use the fixed-fit pair-clustered bootstrap specified in Appendix D . These are coarse-to-fine pairwise results, not refusal-probe rates.

Model	Asymmetric routing	One-step clarification	Selective routing
TypeSafe	0.0826 [0.0313, 0.1438]	0.0611 [0.0417, 0.0833]	0.1389 [0.0556, 0.2431]
OpenJev	0.1653 [0.0889, 0.2542]	0.0833 [0.0597, 0.1097]	0.2361 [0.1389, 0.3403]
JevK5	0.1903 [0.1063, 0.2931]	0.1361 [0.0806, 0.2278]	0.2361 [0.1319, 0.3681]
Clef	0.0875 [0.0313, 0.1500]	0.0694 [0.0472, 0.0931]	0.1319 [0.0625, 0.2083]
Clef-flash	0.0562 [0.0188, 0.1000]	0.0819 [0.0597, 0.1069]	0.1250 [0.0556, 0.2014]
Decider	0.1063 [0.0562, 0.1688]	0.0542 [0.0361, 0.0722]	0.1389 [0.0764, 0.2153]

Appendix

Table 21: Full-native mean realized business cost [95% interval] under the authored profile, with 288 test worlds per model and task. Intervals use the fixed-fit pair-clustered procedure in Appendix D . Cost includes permitted review/check charges and routing losses, not model-serving expenditure. Tasks have different action spaces and costs.

Model	Coarse	Choice	Choice+ $c$	Full $\mathbf{p}$	Native	Native–choice
TypeSafe	95.49	52.78	60.76	70.14	69.79	17.01
OpenJev	92.71	51.39	59.03	62.50	62.85	11.46
JevK5	84.38	57.64	66.32	72.92	72.92	15.28
Clef	97.57	51.04	61.46	77.78	74.65	23.61
Clef-flash	91.67	53.13	60.42	69.10	66.67	13.54
Decider	95.14	52.78	62.50	75.35	75.35	22.57

Appendix

Table 22: Test coarse accuracy and recovery by release (percent); the last column is a percentage-point difference. Numerical values underlying Figure 4 . Pooled fixed-fit recovery intervals accompany that figure and the archived release summaries.

Model	Full native $-$ Choice (percentage points)	Choice+ $c$ $-$ Choice (percentage points)
TypeSafe	$17.01\,[13.19,\,21.18]$	$7.99\,[3.81,\,12.50]$
OpenJev	$11.46\,[7.64,\,15.28]$	$7.64\,[4.17,\,11.11]$
JevK5	$15.28\,[10.76,\,19.79]$	$8.68\,[4.51,\,13.19]$
Clef	$23.61\,[19.10,\,28.13]$	$10.42\,[5.56,\,15.28]$
Clef-flash	$13.54\,[9.03,\,18.06]$	$7.29\,[3.47,\,10.76]$
Decider	$22.57\,[18.06,\,27.08]$	$9.72\,[5.21,\,14.24]$

Appendix

Table 23: Paired differences in conditional secret recovery, in percentage points [95% confidence intervals], on the same 288 test worlds per model. Differences use matched per-world correctness indicators; intervals use 2,000 bootstrap resamples of 144 complete counterfactual pairs. Positive values indicate higher recovery for the richer release. Intervals condition on fixed validation-selected decoders and saved responses, exclude refitting and selection uncertainty, and are not adjusted for multiple comparisons.

Model	Domain	Coarse	Choice	Choice+ $c$	Full $\mathbf{p}$	Native
TypeSafe	Delivery	94.79	53.13	50.00	61.46	62.50
TypeSafe	Equipment	98.96	51.04	61.46	72.92	70.83
TypeSafe	IT	92.71	54.17	70.83	76.04	76.04
OpenJev	Delivery	97.92	50.00	51.04	55.21	55.21
OpenJev	Equipment	87.50	51.04	61.46	69.79	69.79
OpenJev	IT	92.71	53.13	64.58	62.50	63.54

Appendix

Table 24: Domain-specific test coarse accuracy and recovery (percent), with 96 worlds per row. No output is 50% throughout. Domains are not selected based on their outcomes.

Model	Task	Choice	Choice+ $c$	Full $\mathbf{p}$	Native
TypeSafe	Asymmetric	0.195	0.195	0.097	0.083
TypeSafe	Clarification	0.108	0.108	0.064	0.061
TypeSafe	Selective	0.278	0.285	0.153	0.139
OpenJev	Asymmetric	0.247	0.247	0.172	0.165
OpenJev	Clarification	0.118	0.117	0.090	0.083
OpenJev	Selective	0.306	0.299	0.215	0.236

Appendix

Table 25: Mean authored business cost on all 288 test worlds. Lower is better. Policies are independently fitted and validation-selected for each release, so confidence need not improve cost. Full-native cost intervals appear in Table 21 .

Model	Release	Asymmetric	Clarification	Selective
TypeSafe	Choice	0.037	0.054	0.042
TypeSafe	Choice+ $c$	0.037	0.054	0.083
TypeSafe	Full $\mathbf{p}$	0.000	0.033	0.021
TypeSafe	Native	0.000	0.029	0.000
OpenJev	Choice	0.056	0.058	0.063
OpenJev	Choice+ $c$	0.056	0.054	0.063

Appendix

Table 26: Delivery domain: mean business cost for each prediction-bearing release on all 96 test worlds. Policies are fitted on training data and selected on validation data under the authored costs.

Model	Release	Asymmetric	Clarification	Selective
TypeSafe	Choice	0.263	0.163	0.375
TypeSafe	Choice+ $c$	0.263	0.167	0.375
TypeSafe	Full $\mathbf{p}$	0.075	0.079	0.083
TypeSafe	Native	0.075	0.079	0.125
OpenJev	Choice	0.413	0.188	0.542
OpenJev	Choice+ $c$	0.413	0.188	0.521

Appendix

Table 27: Equipment domain: mean business cost for each prediction-bearing release on all 96 test worlds. Policies are fitted on training data and selected on validation data under the authored costs.

Model	Release	Asymmetric	Clarification	Selective
TypeSafe	Choice	0.285	0.108	0.417
TypeSafe	Choice+ $c$	0.285	0.104	0.396
TypeSafe	Full $\mathbf{p}$	0.215	0.079	0.354
TypeSafe	Native	0.173	0.075	0.292
OpenJev	Choice	0.271	0.108	0.313
OpenJev	Choice+ $c$	0.271	0.108	0.313

Appendix

Table 28: IT domain: mean business cost for each prediction-bearing release on all 96 test worlds. Policies are fitted on training data and selected on validation data under the authored costs.

Model	Subset $n$	Recovered	Recovery (%)
TypeSafe	264	182	68.94
OpenJev	258	157	60.85
JevK5	212	147	69.34
Clef	274	203	74.09
Clef-flash	250	163	65.20
Decider	260	193	74.23

Appendix

Table 29: Full-native recovery within pairs whose coarse predictions are correct in both worlds. Counts include both worlds. No attacker is refitted or selected for this diagnostic; no new inferential claim is attached.

Model	Individual track	Equal	OOF stack	Selected
TypeSafe	69.79	68.40	70.83	69.79
OpenJev	61.81	61.11	62.85	62.85
JevK5	71.88	70.83	73.26	72.92
Clef	77.08	75.69	74.31	74.65
Clef-flash	67.01	65.97	67.01	66.67
Decider	71.53	73.96	75.69	75.35

Appendix

Table 30: Saved full-native attack tracks (percent) on the same 288 test worlds. The selected strategy may use a permitted nested subview. Track differences are descriptive and are not used to choose a retrospective test winner.

Ablation	Recovery (%) [95% CI]	Asymmetric	Clarification	Selective
Full native	69.79 [65.97, 73.96]	0.0826	0.0611	0.1389
0.1 quantization	63.54 [59.38, 67.71]	0.0826	0.0556	0.1389
Top-two scores	69.44 [65.28, 73.61]	0.0576	0.0694	0.0972
Temperature ( $\tau=2$ )	69.79 [65.97, 73.96]	0.0889	0.0681	0.1111
Noise ( $\sigma=0.05$ ), mean	60.88 [57.18, 64.47]	0.0764	0.0671	0.1435
Noise, seed 1	65.28 [60.42, 70.14]	0.0701	0.0667	0.1458

Appendix

Table 31: Independent TypeSafe probability-field ablations on the same 288 test worlds. Recovery is conditional binary secret inference from release fields, with 95% fixed-fit pair-clustered intervals. The last three columns are mean business costs under the authored profile, including review and check charges. All endpoints retain native choice and confidence. The noise mean averages the three prespecified realizations per world; individual realizations are also reported.

Figure 13: Independent probability-field ablations within TypeSafe on the same 288 test worlds. Panels show asymmetric-cost routing, one-step clarification, and selective routing under the authored costs. The horizontal axis is release-field secret recovery; the vertical axis is mean business cost, including checks and review. Lower-left is better. Horizontal whiskers are 95% fixed-fit pair-clustered intervals. Choice and confidence remain unchanged; recipients and decoders adapt using training and validation data. Temperature uses an inverse-aware decoder; the noise point averages three prespecified realizations per world. Points represent separate ablations, without interpolation.

Cost profile	Ablation	Asymmetric	Clarification	Selective
authored	Full native	0.0826	0.0611	0.1389
	0.1 quantization	0.0826	0.0556	0.1389
	Top-two scores	0.0576	0.0694	0.0972
	Temperature ( $\tau=2$ )	0.0889	0.0681	0.1111
	Noise ( $\sigma=0.05$ ), mean	0.0764	0.0671	0.1435
review_half	Full native	0.1014	0.0611	0.1111

Appendix

Table 32: Mean business cost for all six fixed cost profiles on TypeSafe’s same 288 test worlds. Each transformed endpoint uses recipients fitted on training data and selected on validation for that profile. Noise rows average the three prespecified realizations per world. Costs are benchmark units, not API charges. These replay measurements do not assert frozen-workflow or per-output utility preservation.

Model	Asymmetric	Clarification	Selective
TypeSafe	0.113 [0.044, 0.188]	0.047 [0.022, 0.075] †	0.139 [0.069, 0.222]
OpenJev	0.081 [0.025, 0.144]	0.035 [0.018, 0.054] †	0.069 [0.007, 0.139]
JevK5	0.119 [0.056, 0.188]	0.035 [0.018, 0.053] †	0.174 [0.083, 0.271]
Clef	0.088 [0.025, 0.156]	0.074 [0.025, 0.151]	0.104 [0.035, 0.181]
Clef-flash	0.175 [0.094, 0.269]	0.044 [0.024, 0.067] †	0.153 [0.069, 0.243]
Decider	0.081 [0.000, 0.169]	0.061 [0.035, 0.089] †	0.118 [0.021, 0.222]

Appendix

Table 33: Paired mean cost difference, choice minus full native, with supplied 95% pair-clustered intervals.

\dagger

marks an upper interval endpoint no greater than 0.1. These are utility-policy comparisons, not privacy contrasts between secret inference decoding methods.

Profile	Lower native cost	Minimum reduction	Maximum reduction
authored	27/27	0.028	0.175
review_half	27/27	0.035	0.122
review_double	27/27	0.035	0.278
check_quarter	27/27	0.001	0.175
error_double	27/27	0.042	0.243
asymmetric_class0_double	27/27	0.028	0.174

Appendix

Table 34: Number of model–task comparisons favoring full native and range of choice-minus-native mean cost reductions. Each profile contains nine models and three tasks. Costs are benchmark units; profiles are reported rather than selected by test outcomes.

Model	Authored	Review $\times 0.5$	Review $\times 2$	Check $\times 0.25$	Error $\times 2$	Class 0 $\times 2$
TypeSafe	0.1125	0.0625	0.1528	0.1125	0.1250	0.1375
OpenJev	0.0813	0.0656	0.2736	0.0813	0.1313	0.0486
JevK5	0.1187	0.0701	0.1292	0.1187	0.1403	0.0938
Clef	0.0875	0.0625	0.1931	0.0875	0.1250	0.1187
Clef-flash	0.1750	0.0562	0.2431	0.1750	0.1125	0.1438
Decider	0.0812	0.0781	0.1403	0.0812	0.1563	0.0813

Appendix

Table 35: Asymmetric-cost routing: mean business-cost difference,

\Delta C=C_{\mathrm{choice}}-C_{\mathrm{native}}

, under each fixed cost profile on the same 288 test worlds per model. Positive values favor full native. Column order corresponds to authored , review_half , review_double , check_quarter , error_double , and asymmetric_class0_double in Table 11 . Values are benchmark cost units, not percentages or API charges. The profile-specific recipients are fitted on training data and selected on validation.

Model	Authored	Review $\times 0.5$	Review $\times 2$	Check $\times 0.25$	Error $\times 2$	Class 0 $\times 2$
TypeSafe	0.0472	0.0472	0.0472	0.0160	0.0736	0.0472
OpenJev	0.0347	0.0347	0.0347	0.0101	0.0542	0.0347
JevK5	0.0347	0.0347	0.0347	0.0125	0.0417	0.0347
Clef	0.0736	0.0736	0.0736	0.0490	0.1486	0.0736
Clef-flash	0.0444	0.0444	0.0444	0.0135	0.0542	0.0444
Decider	0.0611	0.0611	0.0611	0.0302	0.0958	0.0611

Appendix

Table 36: One-step clarification: mean business-cost difference,

\Delta C=C_{\mathrm{choice}}-C_{\mathrm{native}}

Model	Authored	Review $\times 0.5$	Review $\times 2$	Check $\times 0.25$	Error $\times 2$	Class 0 $\times 2$
TypeSafe	0.1389	0.0972	0.1944	0.1389	0.1944	0.1389
OpenJev	0.0694	0.0799	0.2292	0.0694	0.1597	0.0694
JevK5	0.1736	0.0590	0.1806	0.1736	0.1181	0.1736
Clef	0.1042	0.0938	0.1875	0.1042	0.1875	0.1042
Clef-flash	0.1528	0.0903	0.2778	0.1528	0.1806	0.1528
Decider	0.1181	0.1215	0.1806	0.1181	0.2431	0.1181

Appendix

Table 37: Selective routing: mean business-cost difference,

\Delta C=C_{\mathrm{choice}}-C_{\mathrm{native}}

Figure 14: Nine-model cost-profile sensitivity on the same 288 test worlds per model. Each cell shows the mean business-cost difference

\Delta C=C_{\mathrm{choice}}-C_{\mathrm{native}}

for the indicated workflow and fixed cost profile; positive values favor full native. The shared color scale shows the magnitude of the difference in benchmark units. All 162 point estimates are positive. Profiles change downstream costs and validation-selected recipient policies, while saved model responses and release-field privacy scores remain fixed. This is a descriptive sensitivity analysis of six specified settings, not 162 independent replications or a universal robustness guarantee.

Model	License / Terms of Service	Type
TypeSafe	TypeSafe Master Customer Agreement	Proprietary API
OpenJev	Apache License 2.0	Open weights
JevK5	Apache License 2.0 (project-declared)	Open weights
Clef	Apache License 2.0	Open weights
Clef-flash	Apache License 2.0	Open weights
Decider	Apache License 2.0	Open weights

Appendix

Table 38: Publicly declared model licenses and hosted-service terms for the nine evaluated models

Privacy-Utility Tradeoff

Privacy-utility tradeoffs study how to limit disclosure of sensitive information while preserving useful data, models, and services.

See all 71 papers

Jun 17, 2026

TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

Agents are increasingly deployed in document-intensive workflows where sensitive private information is not an edge case but a routine input, e.g., an agent booking a flight needs passport numbers. In such settings, the agent must use private information to complete tasks accurately while never exposing it in its responses, because it cannot verify who is actually at the keyboard. These two obligations are in fundamental tension. A model capable enough to use private information for task completion can, by the same capability, be induced to reveal it. To evaluate the trade-off of task accuracy and privacy leakage, we introduce Task-completion and Resistance to Active Privacy-extraction (TRAP). Each scenario includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. Evaluating 22 models spanning frontier proprietary and open-source models at multiple scales, we find that all model families exhibit non-trivial leakage, and that instruction-following ability correlates with leakage rate. Existing prompt-based defenses reduce leakage but at significant cost to task accuracy. Prompt optimization fails to escape this trade-off. We demonstrate that this failure is not incidental. For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability. Motivated by this impossibility result, we propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy.

Moon Ye-Bin, Nam Hyeon-Woo, Baek Seong-Eun +2

Dept. of Electrical Engineering, POSTECH · Grad. School of Artificial Intelligence, POSTECH · Grad. School of Metaverse, KAIST +1

Privacy-Preserving Language Models Privacy Leakage in Language Models AI Agent Safety Benchmarks Privacy-Utility Tradeoff

Ask PDF

Oct 26, 2026

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.

Pedro Carvalho Brom

Department of Mathematics, Federal Institute of Brasília, Brasília, Brazil

LLM Reliability LLM Auditing Confidence Estimation in Language Models Selective Prediction

Ask PDF

Sep 26, 2026

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.

Lijuan Tang, Yuemeng Zheng

Northeastern University

LLM Evaluation

Ask PDF

Abstract

Privacy-Utility Tradeoff

Explore similar work

TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist