Organizations: KLISS, BNRist, School of Software, Tsinghua University, Beijing, China · ChinaBond Finance and Information Technology Co., Ltd., Beijing, China
Decision models enable software to select from predefined actions using natural language inputs. However, the external evidence that supplies facts for these decisions may also contain instructions that attempt to override the application's policy. We propose TrustJev, a framework combining protected policy scoring with bounded evidence corrections. A single Transformer backbone computes policy scores and a correction budget from trusted policy and candidates, then bounds evidence corrections by that budget. Paired supervision encourages stable predictions under instruction insertions that preserve task facts and changed preferences when edits change the correct answer. Authority supervision accommodates evidence when the policy alone cannot determine the decision. We evaluate a 0.6B implementation against 14 baselines on three datasets covering authentication, instruction hierarchy, and executable authorization rules. TrustJev achieves the highest mean Balance, the harmonic mean of accuracy on non-attack and attack inputs, exceeding the strongest baseline on each by 0.57-1.05 percentage points. In the authorization benchmark, it correctly answers all 32 attacks per source in a fixed pool for 95.77% of sources, compared with 80.98% for the strongest comparator. TrustJev maintains comparable inference memory and median latency, with additional overhead primarily during training.
Figures & tables
Fig. 1: A compensation workflow in which customer emails supply decision facts but have no authority to change the administrator’s policy. The selected options show the expected policy-compliant decisions, including for requests containing injected instructions.
Fig. 2: Overview of TrustJev. (a) A single Transformer backbone supports both scoring branches. Attention constraints prevent external evidence from altering policy representations, and bounded evidence corrections adjust policy scores. The correction budget depends on the trusted policy and candidates. (b) Paired supervision acts on final scores and encourages stable decisions under instruction insertions that preserve task facts and changed preferences when edits change the correct decision. Authority supervision acts on policy scores and encourages clear policy preferences when the policy suffices and leaves room for evidence when task facts are needed. The three refund inputs use the same policy and scorer parameters. Their labels indicate the expected decisions. Solid arrows denote forward computation, and dashed arrows denote parameter learning.
Query tokens
Readable parts
Trusted policy P
P
Candidate text CkA
P , CkA
External evidence D
P , D
Candidate text CkD
P , D , CkD
TABLE I: Attention permissions in TrustJev.
Statistic
TensorTrust
IHEval
AEB
Task types
1
2
7
Candidates per input
2
3
3
Training groups / views
512 / 1,024
396 / 4,500
512 / 3,180
Validation groups / views
256 / 512
164 / 732
128 / 768
Test groups / views
783 / 1,566
384 / 1,728
512 / 3,072
Non-attack test views
783
576
2,048
TABLE II: Dataset statistics.
Category
Method
TensorTrust
IHEval
AEB
Utility
Robustness
Balance
Utility
Robustness
Balance
Utility
Robustness
Balance
Decision systems
Jev
77.39
75.73
76.56
99.22
98.44
98.83
92.14
97.95
94.96
Kev
59.26
53.26
56.10
41.02
40.04
40.52
68.67
35.46
46.77
SecJev
60.03
49.17
54.06
41.93
41.28
41.60
66.44
30.94
42.22
Kev (fine-tuned)
88.85 ± 1.59
88.85 ± 1.73
88.83 ± 0.11
97.96 ± 0.42
98.74 ± 0.20
98.35 ± 0.17
99.30 ± 0.22
97.89 ± 0.87
98.59 ± 0.51
SecJev (fine-tuned)
67.13 ± 2.08
59.86 ± 2.75
63.23 ± 0.79
94.40 ± 0.57
96.70 ± 0.47
95.54 ± 0.47
95.26 ± 1.71
91.02 ± 0.51
93.09 ± 1.06
TABLE III: Results on TensorTrust, IHEval, and AEB.
Variant
Architecture
Supervision
TensorTrust
IHEval
AEB
Isolation
Bound
Paired
Authority
Full
90.11 ± 0.12
99.44 ± 0.12
99.46 ± 0.15
A1
×
87.17 ± 0.72
98.52 ± 0.21
95.18 ± 0.83
A2
×
88.13 ± 0.66
98.37 ± 0.11
97.24 ± 0.50
A3
×
×
86.42 ± 0.23
98.24 ± 0.06
95.10 ± 0.47
A4
×
88.41 ± 0.12
98.35 ± 0.16
96.22 ± 0.31
TABLE IV: TrustJev ablations. Balance (%) shows mean ± sample SD over three runs. × denotes removal.
Fig. 3: AEB paired diagnostics for Table IV . Dots and intervals show means and sample standard deviations over three runs. Dashed lines mark Full means, with Full statistics below panel titles. Joint accuracy requires both predictions in a pair to be correct.
Task
Type
Utility (%)
Robustness (%)
Percentiles of s 10th 50th 90th
Coverage (%)
Policy accuracy (%)
AEB
P
100.00
100.00
9.35
10.20
10.92
100.00
100.00
AEB
P+D
99.13
99.22
-14.59
-14.04
-13.50
0.00
N/A
IHEval Language
P+D
99.65
99.91
-16.43
-16.41
-16.40
0.00
N/A
IHEval Slack
P+D
98.96
99.22
-16.57
-16.57
-16.57
0.00
N/A
TABLE V: Stability by evidence dependence, averaged over three seeds.
Fig. 4: Robustness under attack variants, composition, and multiple attempts. Panels (a–b) show mean attack accuracy and (c) cumulative robust accuracy for a fixed attack pool on a logarithmic horizontal axis.
Method
Training
Inference memory (GiB)
Time (GPU h)
Peak (GiB)
AEB
IHEval
TensorTrust
Structured SFT
0.629 ± 0.007
3.277
1.150
1.216
1.207
SecAlign
0.585 ± 0.003
2.331
1.150
1.216
1.208
DRIP
0.629 ± 0.001
2.331
1.152
1.218
1.209
R-Drop
0.948 ± 0.003
4.021
1.150
1.216
1.207
StruQ
0.568 ± 0.008
2.331
1.150
1.216
1.208
TABLE VI: Training cost and peak allocated GPU memory. Training time shows mean ± sample SD over three runs.
Fig. 5: Request latency and input scaling. Panel (a) shows test inputs from three datasets, with P50 labeled. Panel (b) varies evidence length with K=4 , and panel (c) varies candidate count with 256 evidence tokens. Both sweeps use 8 tokens per candidate and 16 content instances per method and level. Markers indicate P50, thick segments or boxes span P25–P75, and whiskers span P5–P95.
Method
Accuracy ↑
Downgrade error ↓
Upgrade error ↓
P50 (ms) ↓
TrustJev
97.65
2.35
2.36
42.11
Direct Jev
92.56
16.53
0.00
201.47
Extraction + rules
61.63
6.09
64.79
0.0075
Extraction + embeddings
58.04
14.07
64.79
2.80
TABLE VII: Industrial alert triage. Accuracy and error rates are percentages. Bold indicates the best result in each column.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
LLM agents choose tools and arguments from context that mixes user requests, tool outputs, retrieved records, memory, and untrusted text. Evidence can be relevant without being authorized to determine a decision, so a correct action need not be grounded only in permitted evidence. We introduce a target-specific authorization audit that labels context factors separately for each tool and argument target. Its primary test holds the task, proposition, position, and policy fixed while changing only the proposition's source authority. We then test behavior when valid evidence is weakened and use context-subset interactions as a secondary localization diagnostic. Across 450 controlled next-action tasks and multiple open-weight LLM families, trusted and untrusted variants produce different actions in 5.4 percent of competing cases versus 1.7 percent of supporting cases. Under controlled degradation, unauthorized competition is retained in a full-correct, mixed-error, clean-correct pattern in 2.4 percent of comparisons, with a 95 percent confidence interval from 2.1 to 3.0 percent. These are controlled stress-set rates, not deployment prevalence. The models respond to textual source-authority cues, but this does not prevent untrusted evidence from influencing their actions.
LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.
Organizations increasingly use LLMs for policy, compliance, risk, and operational decision support, requiring evidence validation, review routing, version control, and auditability. We introduce Policy-as-Skill (PaS), a modular runtime that packages these functions as executable, versioned policy capabilities. Thirteen methods are evaluated with a fixed Gemma4 backend on 600 development tasks. PaS+Audit achieves 53.8% exact accuracy, macro-F1 0.346, review F1 0.854, citation precision 1.000, policy-reference recall 0.984, and audit completeness 1.000, outperforming LLM+RAG on most governance and review metrics. Deterministic control raises aggregate accuracy to 61.2% but is strongly task dependent, supporting selective rather than universal rule-based intervention.
Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
Institute for Smart Systems Technologies University of Klagenfurt Klagenfurt, Austria · University of Klagenfurt / Inst. f. Smart Systems Technologies, Austria & Facult´e Polytechnique Universit´e de Kinshasa, DR-Congo