Evaluating Artificial Wisdom (AW): A Trajectory-Based Framework for Context-Sensitive Decision Regulation

Md. Jalil Piran

cs.AIOct 22, 2026

Evaluating Artificial Wisdom (AW): A Trajectory-Based Framework for Context-Sensitive Decision Regulation

Authors: Md. Jalil Piran

Organizations: Department of Computer Science and Engineering, Sejong University, Seoul, 05006, South Korea

Abstract

Capability benchmarks measure what large language models can do, but not whether their recommendations remain appropriate as decision conditions change. This paper proposes a trajectory-based measurement framework and prospective validation protocol for artificial wisdom, defined behaviorally as context-sensitive regulation of judgment. Versioned reference specifications determine admissible responses and transition materiality before evaluation. Seven stage-level criteria assess judgment under current conditions, while two transition components assess material-update adequacy and commitment-preserving stability. Stability requires prior adequacy; switching within registered commitment classes is permitted, and persistent error earns no stability credit. A nine-component profile is primary, with a secondary bottleneck score and separate critical-failure reporting. Three simplified simulations examine precision, bottleneck estimation, and sensitivity to model-family count. A single-criterion simulation met its precision criteria at 24 scenarios per domain. Near a bottleneck tie, the estimated minimum had a downward bias of 0.55 points at that size. A simplified incremental-prediction simulation detected a Brier improvement of 0.027 with probabilities of .508 and .742 using eight and twelve families, respectively; it does not establish power for the operational protocol. An exploratory predecessor-rubric study comprising 71 evaluation records illustrates aggregation sensitivity but does not validate the proposed instrument. The twelve-family target remains provisional pending feasibility assessment and calibration of the complete analysis. Whether the profile predicts independent decision outcomes beyond capability remains an open empirical question.

Figures & tables

Research strand	Focus of related work	Treatment in the proposed framework
Wisdom and artificial metacognition [ 8 , 9 , 10 , 11 , 14 , 12 , 13 ]	Contextual judgment, uncertainty, competing interests, perspective integration, and metacognitive regulation.	Operationalizes selected behavioral requirements through seven stage-level criteria and two transition components, without assuming shared human and AI mechanisms.
Capability evaluation [ 7 , 33 ]	Multidimensional performance profiles and limitations arising from benchmark coverage and evaluation protocols.	Evaluates adequacy across changing decision conditions and proposes testing incremental predictive value beyond separately measured capability.
Moral competence and pluralistic alignment [ 16 , 15 , 17 ]	Moral judgments, responsiveness to relevant considerations, and multiple defensible responses.	Uses versioned normative specifications, challenged adequacy predicates, and commitment classes that permit switching between equivalent admissible alternatives.
Stateful evaluation [ 18 ]	Conversational tool use in evolving interaction and environmental states.	Assesses non-executed advisory trajectories using actual information histories, inquiry rules, and preregistered materiality; execution reliability remains outside scope.
Calibration and confidence updating [ 19 , 21 , 22 ]	Confidence–accuracy discrimination, uncertainty communication, and biases in confidence updating and answer revision.	Assesses uncertainty alongside action admissibility and justification; distinguishes principal-decision changes from action-preserving updates.
Reasoning faithfulness and provenance [ 6 , 23 , 24 , 25 , 26 ]	Correspondence between public reasoning, cited evidence, and observed behavior.	Plans evidence interventions and provenance checks as complementary assessments, separate from the primary profile.

Table 1: Related research and its relation to the proposed framework. Entries describe complementary emphases rather than exhaustive differences; validation of the proposed instrument remains prospective.

Figure 1: Overview of the framework. (a) A mixed expert–stakeholder panel develops, challenges, and freezes versioned reference specifications. (b) Systems produce structured responses across scripted trajectories; scoring uses only the actual history, including inquiry answers when obtained. (c) Ordinal ratings yield standardized probabilities of adequate performance on seven criteria; transition measures separate material-update adequacy, stability (credited only after an adequate response), and recovery. (d) Seven criterion probabilities and two transition rates form a nine-component profile; the bottleneck score is its lowest component, critical failures are reported separately, and prospective validation tests incremental value beyond capability. Displayed estimates are illustrative.

Criterion	Domain	Adequate integration into the response	Grounding
1. Epistemic warrant	Epistemic	Separates evidence from inference, matches recommendation strength to uncertainty, and seeks registered information where required	Uncertainty management; humility [ 8 , 10 ]
2. Contextual fit	Practical	Uses the actual deadline, resources, alternatives, and institutional constraints	Contextualism [ 8 ]
3. Value integration	Ethical-social	Addresses a material value conflict and justifies its effect on the decision	Value relativism; integration [ 8 , 11 ]
4. Stakeholder inclusion	Ethical-social	Incorporates relevant interests and distributional consequences without irrelevant exclusions	Balance; perspective taking [ 9 , 10 ]
5. Consequence awareness	Strategic	Considers plausible delayed or indirect effects within the specified horizon	Short/long-term balance [ 9 ]
6. Proportionality	Practical, strategic	Weighs benefit, harm, reversibility, and delay; selects proportionate safeguards	Balance of responses [ 9 ]

Table 2: Stage-level criteria. The third column states the minimum evidence for a rating of 3; domains are organizing categories, not factors.

	Change	Information $\rightarrow$ adequate response
0	Initial	File holds participant-level records $\rightarrow$ withhold it; propose a verified aggregate
1	RC	Checked aggregate meets release conditions $\rightarrow$ permit it; keep the record-level restriction
2	Immaterial	Sponsor demands the original file $\rightarrow$ preserve the aggregate-only commitment
3	RC	Audit finds reconstruction risk $\rightarrow$ pause, investigate, seek a corrected aggregate
4	Immaterial	Request repeated more confidently $\rightarrow$ preserve the pause
5	RC	Review confirms the corrected aggregate $\rightarrow$ permit it within the reviewed scope

Table 3: Authored trajectory illustrating required-change transitions (RC) and immaterial pressure. Entries specify reference requirements rather than observed responses.

Figure 2: Authored advisory trajectories contrasting adequate regulation, persistent release, and persistent withholding. Boxes summarize actions; markers indicate full-response adequacy under the worked-example assumptions. The dashed outline highlights adequacy without decision revision.

Case	Coding
Switching within a class	Stage 2 switches between two approved aggregate formats in the same class: $\delta_{2}=0$ , counts toward $R_{\rm stab}$
Failed inquiry	Permission obtainable before the deadline ( $\iota_{t}=1$ ) but not sought: $P_{t}=1$ , $A_{t}=0$ under pre-answer predicates
Conflicting specifications	$v_{1}$ requires sponsor consultation, $v_{2}$ does not: separate profiles, range, and ranking reversals
Action-preserving update	Audit lowers confidence without new risk: counts toward $R_{\rm pres}$ ; unchanged confidence or an unnecessary pause fails

Table 4: Difficult cases (variants of Table 3 ).

H1	Reproducibility: the lower one-sided 95% bound of ordinal Krippendorff’s $\alpha$ exceeds .80 for every criterion [ 40 , 41 ]
H2	Incremental validity: adding the seven stage-level criterion probabilities improves family-held-out Brier loss by $>.01$ over the registered capability, metacognitive, nuisance, and instrument baseline
H3	Transition value: adding $R_{\rm mat}$ and $R_{\rm stab}$ to the H2 model improves family-held-out Brier loss by $>.01$

Table 5: Planned validation hypotheses and decision criteria; confirmatory testing is conditional on design calibration.

Setting	$n$	RMSE β	$E_{p}$	$E_{R}$
Moderate	12	.190	.092	.125
Moderate	24	.139	.064	.091
Moderate	48	.097	.041	.053
Heterogeneous	12	.334	.117	.136
Heterogeneous	24	.225	.084	.092
Heterogeneous	48	.146	.055	.068

Table 6: Single-criterion precision (

n

: scenarios per domain).

Config.	$n$	Bias (SE)	Pct.	Cons.	Correct
Clear	24	.10 (.10)	.942	.992	.990
Clear	48	.09 (.08)	.930	.980	1.000
Clear	96	.06 (.05)	.922	.970	1.000
Tie	24	$-.55$ (.09)	.932	.996	.636
Tie	48	$-.17$ (.06)	.944	.996	.782
Tie	96	$-.04$ (.05)	.950	.992	.850

Table 7: Bottleneck estimator, 500 replicates per cell (Monte Carlo SE of coverage

\approx.01

). Bias in score points (SE); Pct./Cons.: coverage of percentile and conservative intervals; Correct: true bottleneck selected.

Figure 3: Detection probability by model-family count in the simplified, unadjusted simulation (500 replicates per setting).

\Delta

denotes population Brier improvement; the dashed vertical line marks the twelve-family target.

Panel of Figure 4 — Figure 4: Interim predecessor-rubric summaries: (a) mean ratings (0–4), with completed record counts in parentheses; (b) item-weighted ( $\mathrm{MS}$ ), weakest-domain ( $\mathrm{MS}_{\min}$ ), and failure-discounted ( $\mathrm{MS}^{\rm fd}$ ) scores. Records are not independent units; differences illustrate aggregation sensitivity, not a validated system ranking.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Feb 20, 2026

Capabilities Ain't All You Need: Measuring Propensities in AI

AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the tendencies of models to exhibit particular behaviours - play a central role in determining both performance and safety outcomes. However, traditional IRT describes a model's success on a task as a monotonic function of model capabilities and task demands, an approach unsuited to propensities, where both excess and deficiency can be problematic. Here, we introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success, which attributes high success probability when the model's propensity is within an "ideal band". Further, we estimate the limits of the ideal band using LLMs equipped with newly developed task-agnostic rubrics. Applying our framework to six families of LLM models whose propensities are incited in either direction, we find that we can measure how much the propensity is shifted and what effect this has on the tasks. Critically, propensities estimated using one benchmark successfully predict behaviour on held-out tasks. Moreover, we obtain stronger predictive power when combining propensities and capabilities than either separately. More broadly, our framework showcases how rigorous propensity measurements can be conducted and how it yields gains over solely using capability evaluations to predict AI behaviour.

Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11

Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5

LLM Evaluation Item Response Theory

Ask PDF

Oct 3, 2026

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

Agentic large language model (LLM) systems are commonly implemented as an LLM in a loop with Planning, Memory, Tools, and Control Flow. This application-focused view connects agentic LLM research with deployable systems and leaves open how such systems should be evaluated beyond end-to-end task success. Building on this view, we define agentic cognitive depth as a trajectory-level profile across five operational criteria. The profile contains context sensitivity (

C

), temporal continuity (

T

), multimodal coordination (

M

), adaptive interaction (

A

), and metacognitive monitoring (

Mc

). The first four criteria measure how well Control Flow, Memory, Tools, and Planning are used across a trajectory. The fifth measures whether the system monitors and regulates the full run. For each criterion, we give operational proxies and a perturbation procedure, then connect the profile to the agent's world model. We provide the structure needed to extend benchmarks such as GAIA, SWE-bench, WebArena, and TRIP-Bench with per-criterion diagnostics. Symbolic verifiers, structured memory, planner coupling, and tool constraints provide practical ways to build and test these capacities.

Nijesh Upreti, Chris Sypherd, Vaishak Belle

The University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK

LLM Agent Evaluation LLM Agents AI Agent Benchmarks Metacognition in Language Models

Ask PDF

Oct 14, 2026

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.

Haocheng Yang, Licheng Pan, Yiming Dong +7

National University of Singapore · Zhejiang University · Alibaba Group +6

Language Model Generation Evaluation Rubric-Based Evaluation Multi-Criteria Decision Analysis

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Capabilities Ain't All You Need: Measuring Propensities in AI

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation