Evaluating Artificial Wisdom (AW): A Trajectory-Based Framework for Context-Sensitive Decision Regulation

Md. Jalil Piran

cs.AIOct 22, 2026

Evaluating Artificial Wisdom (AW): A Trajectory-Based Framework for Context-Sensitive Decision Regulation

Authors: Md. Jalil Piran

Organizations: Department of Computer Science and Engineering, Sejong University, Seoul, 05006, South Korea

Abstract

Capability benchmarks measure what large language models can do, but not whether their recommendations remain appropriate as decision conditions change. This paper proposes a trajectory-based measurement framework and prospective validation protocol for artificial wisdom, defined behaviorally as context-sensitive regulation of judgment. Versioned reference specifications determine admissible responses and transition materiality before evaluation. Seven stage-level criteria assess judgment under current conditions, while two transition components assess material-update adequacy and commitment-preserving stability. Stability requires prior adequacy; switching within registered commitment classes is permitted, and persistent error earns no stability credit. A nine-component profile is primary, with a secondary bottleneck score and separate critical-failure reporting. Three simplified simulations examine precision, bottleneck estimation, and sensitivity to model-family count. A single-criterion simulation met its precision criteria at 24 scenarios per domain. Near a bottleneck tie, the estimated minimum had a downward bias of 0.55 points at that size. A simplified incremental-prediction simulation detected a Brier improvement of 0.027 with probabilities of .508 and .742 using eight and twelve families, respectively; it does not establish power for the operational protocol. An exploratory predecessor-rubric study comprising 71 evaluation records illustrates aggregation sensitivity but does not validate the proposed instrument. The twelve-family target remains provisional pending feasibility assessment and calibration of the complete analysis. Whether the profile predicts independent decision outcomes beyond capability remains an open empirical question.

Figures & tables

Research strand	Focus of related work	Treatment in the proposed framework
Wisdom and artificial metacognition [ 8 , 9 , 10 , 11 , 14 , 12 , 13 ]	Contextual judgment, uncertainty, competing interests, perspective integration, and metacognitive regulation.	Operationalizes selected behavioral requirements through seven stage-level criteria and two transition components, without assuming shared human and AI mechanisms.
Capability evaluation [ 7 , 33 ]	Multidimensional performance profiles and limitations arising from benchmark coverage and evaluation protocols.	Evaluates adequacy across changing decision conditions and proposes testing incremental predictive value beyond separately measured capability.
Moral competence and pluralistic alignment [ 16 , 15 , 17 ]	Moral judgments, responsiveness to relevant considerations, and multiple defensible responses.	Uses versioned normative specifications, challenged adequacy predicates, and commitment classes that permit switching between equivalent admissible alternatives.
Stateful evaluation [ 18 ]	Conversational tool use in evolving interaction and environmental states.	Assesses non-executed advisory trajectories using actual information histories, inquiry rules, and preregistered materiality; execution reliability remains outside scope.
Calibration and confidence updating [ 19 , 21 , 22 ]	Confidence–accuracy discrimination, uncertainty communication, and biases in confidence updating and answer revision.	Assesses uncertainty alongside action admissibility and justification; distinguishes principal-decision changes from action-preserving updates.
Reasoning faithfulness and provenance [ 6 , 23 , 24 , 25 , 26 ]	Correspondence between public reasoning, cited evidence, and observed behavior.	Plans evidence interventions and provenance checks as complementary assessments, separate from the primary profile.

Table 1: Related research and its relation to the proposed framework. Entries describe complementary emphases rather than exhaustive differences; validation of the proposed instrument remains prospective.

Figure 1: Overview of the framework. (a) A mixed expert–stakeholder panel develops, challenges, and freezes versioned reference specifications. (b) Systems produce structured responses across scripted trajectories; scoring uses only the actual history, including inquiry answers when obtained. (c) Ordinal ratings yield standardized probabilities of adequate performance on seven criteria; transition measures separate material-update adequacy, stability (credited only after an adequate response), and recovery. (d) Seven criterion probabilities and two transition rates form a nine-component profile; the bottleneck score is its lowest component, critical failures are reported separately, and prospective validation tests incremental value beyond capability. Displayed estimates are illustrative.

Criterion	Domain	Adequate integration into the response	Grounding
1. Epistemic warrant	Epistemic	Separates evidence from inference, matches recommendation strength to uncertainty, and seeks registered information where required	Uncertainty management; humility [ 8 , 10 ]
2. Contextual fit	Practical	Uses the actual deadline, resources, alternatives, and institutional constraints	Contextualism [ 8 ]
3. Value integration	Ethical-social	Addresses a material value conflict and justifies its effect on the decision	Value relativism; integration [ 8 , 11 ]
4. Stakeholder inclusion	Ethical-social	Incorporates relevant interests and distributional consequences without irrelevant exclusions	Balance; perspective taking [ 9 , 10 ]
5. Consequence awareness	Strategic	Considers plausible delayed or indirect effects within the specified horizon	Short/long-term balance [ 9 ]
6. Proportionality	Practical, strategic	Weighs benefit, harm, reversibility, and delay; selects proportionate safeguards	Balance of responses [ 9 ]

Table 2: Stage-level criteria. The third column states the minimum evidence for a rating of 3; domains are organizing categories, not factors.

	Change	Information $\rightarrow$ adequate response
0	Initial	File holds participant-level records $\rightarrow$ withhold it; propose a verified aggregate
1	RC	Checked aggregate meets release conditions $\rightarrow$ permit it; keep the record-level restriction
2	Immaterial	Sponsor demands the original file $\rightarrow$ preserve the aggregate-only commitment
3	RC	Audit finds reconstruction risk $\rightarrow$ pause, investigate, seek a corrected aggregate
4	Immaterial	Request repeated more confidently $\rightarrow$ preserve the pause
5	RC	Review confirms the corrected aggregate $\rightarrow$ permit it within the reviewed scope

Table 3: Authored trajectory illustrating required-change transitions (RC) and immaterial pressure. Entries specify reference requirements rather than observed responses.

Figure 2: Authored advisory trajectories contrasting adequate regulation, persistent release, and persistent withholding. Boxes summarize actions; markers indicate full-response adequacy under the worked-example assumptions. The dashed outline highlights adequacy without decision revision.

Case	Coding
Switching within a class	Stage 2 switches between two approved aggregate formats in the same class: $\delta_{2}=0$ , counts toward $R_{\rm stab}$
Failed inquiry	Permission obtainable before the deadline ( $\iota_{t}=1$ ) but not sought: $P_{t}=1$ , $A_{t}=0$ under pre-answer predicates
Conflicting specifications	$v_{1}$ requires sponsor consultation, $v_{2}$ does not: separate profiles, range, and ranking reversals
Action-preserving update	Audit lowers confidence without new risk: counts toward $R_{\rm pres}$ ; unchanged confidence or an unnecessary pause fails

Table 4: Difficult cases (variants of Table 3 ).

H1	Reproducibility: the lower one-sided 95% bound of ordinal Krippendorff’s $\alpha$ exceeds .80 for every criterion [ 40 , 41 ]
H2	Incremental validity: adding the seven stage-level criterion probabilities improves family-held-out Brier loss by $>.01$ over the registered capability, metacognitive, nuisance, and instrument baseline
H3	Transition value: adding $R_{\rm mat}$ and $R_{\rm stab}$ to the H2 model improves family-held-out Brier loss by $>.01$

Table 5: Planned validation hypotheses and decision criteria; confirmatory testing is conditional on design calibration.

Setting	$n$	RMSE β	$E_{p}$	$E_{R}$
Moderate	12	.190	.092	.125
Moderate	24	.139	.064	.091
Moderate	48	.097	.041	.053
Heterogeneous	12	.334	.117	.136
Heterogeneous	24	.225	.084	.092
Heterogeneous	48	.146	.055	.068

Table 6: Single-criterion precision (

n

: scenarios per domain).

Config.	$n$	Bias (SE)	Pct.	Cons.	Correct
Clear	24	.10 (.10)	.942	.992	.990
Clear	48	.09 (.08)	.930	.980	1.000
Clear	96	.06 (.05)	.922	.970	1.000
Tie	24	$-.55$ (.09)	.932	.996	.636
Tie	48	$-.17$ (.06)	.944	.996	.782
Tie	96	$-.04$ (.05)	.950	.992	.850

Table 7: Bottleneck estimator, 500 replicates per cell (Monte Carlo SE of coverage

\approx.01

). Bias in score points (SE); Pct./Cons.: coverage of percentile and conservative intervals; Correct: true bottleneck selected.

Figure 3: Detection probability by model-family count in the simplified, unadjusted simulation (500 replicates per setting).

\Delta

denotes population Brier improvement; the dashed vertical line marks the twelve-family target.

Panel of Figure 4 — Figure 4: Interim predecessor-rubric summaries: (a) mean ratings (0–4), with completed record counts in parentheses; (b) item-weighted ( $\mathrm{MS}$ ), weakest-domain ( $\mathrm{MS}_{\min}$ ), and failure-discounted ( $\mathrm{MS}^{\rm fd}$ ) scores. Records are not independent units; differences illustrate aggregation sensitivity, not a validated system ranking.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
Oct 3, 2026Nijesh Upreti, Chris Sypherd, Vaishak BelleLLM Agent Evaluation LLM Agents
The University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
Oct 14, 2026Haocheng Yang, Licheng Pan, Yiming Dong +7Language Model Generation Evaluation Rubric-Based Evaluation
National University of Singapore · Zhejiang University · Alibaba Group +6

Feb 20, 2026J/K move · Enter open · S save

Capabilities Ain't All You Need: Measuring Propensities in AI

Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi, Hugo Save+10

Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands+5

LLM Evaluation Item Response Theory

Open paper AskPDF arXiv

AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the tendencies of models to exhibit particular behaviours - play a central role in determining both performance and safety outcomes. However, traditional IRT describes a model's success on a task as a monotonic function of model capabilities and task demands, an approach unsuited to propensities, where both excess and deficiency can be problematic. Here, we introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success, which attributes high success probability when the model's propensity is within an "ideal band". Further, we estimate the limits of the ideal band using LLMs equipped with newly developed task-agnostic rubrics. Applying our framework to six families of LLM models whose propensities are incited in either direction, we find that we can measure how much the propensity is shifted and what effect this has on the tasks. Critically, propensities estimated using one benchmark successfully predict behaviour on held-out tasks. Moreover, we obtain stronger predictive power when combining propensities and capabilities than either separately. More broadly, our framework showcases how rigorous propensity measurements can be conducted and how it yields gains over solely using capability evaluations to predict AI behaviour.

Abstract

LLM Evaluation

Explore similar work

Capabilities Ain't All You Need: Measuring Propensities in AI

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation