Policy Evaluation

Follow this topic Ask about this topic

Policy evaluation asks how well a decision rule performs and whether it improves on alternatives in settings such as reinforcement learning, marketplaces, healthcare, and public policy. Common approaches include temporal-difference and off-policy value estimation, propensity-score and inverse-probability methods, randomized or adaptive experiments, and simulation with agent-based models or digital twins. Researchers also develop confidence intervals, risk-sensitive measures, and benchmarks to compare policies when observations are limited or outcomes are uncertain. A central challenge is that offline estimates and benchmark rankings can mislead under confounding, distribution shift, or adaptive behavior, so reliable assessment must connect metrics to deployment goals and often requires prospective validation.

Latest papers

55 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

Jul 25, 2026

Online Policy Evaluation for MDPs with Dynamic UBSR Measures

Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.

Weikai Wang, Erick Delage

GERAD & Department of Decision Sciences, HEC Montr´eal, Canada

Policy Evaluation Risk-Sensitive RL Temporal-Difference Learning

Jul 17, 2026

A Formally Grounded ODRL Evaluator: Implementation and Comparison

The ODRL policy language is emerging as the de-facto standard for policy modelling data access and usage preferences, AI governance policies and data workflows in European dataspaces. The current standard has no mathematical formal semantics to describe how a system should implement policy evaluation. This has resulted in a variety of systems and tools that implement their own interpretation of the language, which limits interoperability and cannot guarantee consistent results. Based on an existing semantic model of ODRL, we formalise the problems of ODRL evaluation for the access control and monitoring scenarios, in both static and streaming settings, and we provide a novel, efficient algorithm and implementation. We present the first ODRL Evaluator with transparent formal semantics and supporting all rule types. We experimentally measure its performance, analysing different scalability dimensions related to policy complexity and size of the data on which a policy is evaluated. We compare our system with the state-of-the-art by providing a comparative review of existing ODRL evaluators, which highlights the differences in supported ODRL features and evaluation modes.

Jaime Osvaldo Salas, Paolo Pareti, Adeel Aslam +2

University of Southampton, University Road, Southampton, SO17 1BJ, UK

Policy Evaluation Access Control

Jul 11, 2026

Comparing Socially-Equitable Renewable Energy Budget Allocation MDP Policies in Mature and Emerging Economies

Equitable renewable-energy planning is a sequential decision problem, but the decision variables available to a public planner differ sharply between mature and emerging economies. In the former the government largely builds generation, while in the latter it steers private investment through incentives and quotas. We formulate socially-equitable renewable-energy budget allocation as a Markov Decision Process (MDP) and, using a single problem-agnostic solver interface, compare the same policies across the two settings: eight U.S. cities (a mature economy) and West Java, Indonesia (an emerging economy). The results show that across both settings, a receding-horizon value-iteration policy dominates. In the U.S., it reaches 66% renewable penetration while cutting the underserved low-income population by 96% versus a random baseline. In West Java it closes the low-access gap while crowding in about seven times the private capital of a naive market-chasing heuristic. More interestingly, this heuristic, which is mildly sub-optimal in the U.S., could yield catastrophic outcomes in Indonesia, by underserving every low-access region, because chasing attractive markets and serving the underserved diverge once the planner acts through private developers.

Riya Kinnarkar, Mansur M. Arief, Yan Pratama Akhra +1

Management & Technology University of Pennsylvania Philadelphia, PA, USA

Fair Resource Allocation Policy Evaluation Power Systems

Jul 3, 2026

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with paired gap (LLM minus benchmark) -0.276 and 95% CI [-0.692, 0.146]. A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: non-semantic proposers with up to 2,500 evaluations fall 8.77 - 14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91. These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.

Peiying Zhu, Sidi Chang

Blossom AI San Francisco, USA · Blossom AI Labs Tokyo, Japan

LLM Agent Evaluation Policy Evaluation Language Model Auditing

Jun 29, 2026

Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

Real-world evaluation is the gold standard for robot manipulation policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on robot imitation learning policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE uses a zero-shot agent to localize task-critical segments, restricts error computation to those segments, and applies simple action-alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE generally yields a stronger correlation between validation error and rollout performance than raw MSE. Across 27 distinct policy checkpoints, CI-MSE achieves a Spearman rank correlation of

-0.87

, closer to the ideal value of

-1

than raw MSE's

-0.61

, demonstrating a significant improvement. Sensitivity analysis shows that our metric is robust across a wide range of hyperparameter settings. We further study the effectiveness of CI-MSE under evaluation distribution shifts and identify practical boundaries for its use. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: https://ci-mse.github.io/

Haoxu Huang, Tongsam Zheng, Yifan Chen +2

Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

Robot Policy Learning Off-Policy Evaluation Robotic Policy Evaluation Policy Evaluation Robotic Manipulation

Jun 27, 2026

Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy

Metric aggregation divergence (MAD) is the silent inconsistency that arises when distinct pipeline stages in an agent-based model coupled with a multi-objective evolutionary algorithm (ABM+MOEA) independently re-implement how an outcome metric is extracted from simulation trajectories. Unlike deliberate analytical choices, MAD operates at the level of pipeline architecture: each stage is internally coherent, and the inconsistency becomes visible only when cross-stage outputs are compared. Code inspection of EpidemiOptim, a JAIR-published epidemic policy toolbox, reveals three structurally independent aggregation paths in peer-reviewed code. A faithful replication of this structure produces champion disagreement in 64.2% of independent runs (n=500, 95% CI: [59.9%, 68.3%]). In a 300-seed policy-flip experiment, divergent aggregation causes the optimizer to recommend the wrong champion in 83% of replications, with a mean welfare gap of 2.19 units and a Gini inequality gap of 0.050 units. In a follow-up inference audit, 3 of 249 flipped seeds cross the significance boundary itself. A complementary enterprise follow-up produces the predicted null under near-commensurable rankings (rho = 0.991), while a public upstream rerun of the Lake Problem DPS workflow shows that the archived published-path recommendation reaches joint-threshold success 0.401 whereas a shared contract-path rule reaches 0.552. We introduce the metric contract - a single shared callable enforced at dispatch time across all pipeline stages - as the remedy. Framed as standard engineering discipline applied to the cross-stage metric interface, the contract eliminates divergence by construction with approximately 3% runtime overhead.

Ruiyu Zhang, Lin Nie, Xin Zhao

Department of Politics and Public Administration · Department of Applied Social Sciences

Software Engineering Policy Evaluation Agent-Based Modeling

Jun 23, 2026

Minimax PAC Bounds for Learning in Exogenous Contextual MDPs

We introduce a PAC framework in which the learner can access sampling oracles both before and at decision time. Sample complexity is measured by a pair

(n,m)

, where

n

is the learning budget spent before a query is known and

m

is the additional sampling budget per query. We demonstrate its relevance in discounted Markov decision processes with exogenous i.i.d.\ contexts revealed before acting. Contexts may affect both rewards and transitions but remain uncontrolled by the agent. The learner can sample the unknown context distribution and the transition kernel. We study policy evaluation (PE), best-value estimation (BVE), and best-policy extraction (BPE). When rewards and transitions are known, a variance-reduced algorithm solves all three tasks with sample complexity

\bigl(\widetilde O((1-γ)^{-3}\varepsilon^{-2}),0\bigr)

, which is minimax optimal up to logarithmic factors. Let

\mathcal{X}

be the controlled state space. When transitions are also unknown, we give a PE algorithm with complexity

\bigl(\widetilde O(|\mathcal X|(1-γ)^{-3}\varepsilon^{-2}), \widetilde O((1-γ)^{-2}\varepsilon^{-2})\bigr)

and matching lower bounds at this budget pair. For BVE and BPE, we give an algorithm with a common offline budget

\widetilde O(|\mathcal X|^2|\mathcal A|(1-γ)^{-4}\varepsilon^{-2})

and respective query costs

\widetilde O(|\mathcal A|(1-γ)^{-2}\varepsilon^{-2})

and

\widetilde O(|\mathcal A|(1-γ)^{-3}\varepsilon^{-2})

. Importantly, all bounds are independent of the context-space cardinality.

Corentin Pla, Hugo Richard, Marc Abeille +1

CREST, ENSAE · Criteo AI Lab · FairPlay Joint Team

Markov Decision Processes Reinforcement Learning Policy Evaluation Interval Markov Decision Processes Policy Learning Sample-Efficient RL

Jun 21, 2026

Fed-CausalDiff: Decoupled Synchronization for Federated Do-Simulation and Policy Evaluation

While federated learning enables collaborative modelling on decentralised data, standard methods merely fit historical observations. This purely observational approach is fundamentally insufficient for interventional inference and policy evaluation, as sequential actions dynamically alter future states. We propose Fed-CausalDiff, a federated causal diffusion framework for do-simulation. The architecture decomposes the evolution of the latent state into a global causal score function and a local confounding score function. This design enables decoupled synchronisation (DSS), where clients aggregate only the shared causal mechanism while retaining site-specific confounders locally to handle heterogeneity. Experiments on four datasets demonstrate that Fed-CausalDiff achieves better ATE and policy-value estimation accuracy, offering a favorable trade-off between communication cost and inference fidelity.

Pengfei Li, Mohammad Khalil

Centre for the Science of Learning & Technology (SLATE), University of Bergen Bergen, Norway

Policy Evaluation Causal Inference Diffusion Models Heterogeneous Federated Learning

Jun 20, 2026

Causal Gaussian Processes for Robust Treatment Effect Evaluation with Unobserved Confounding

The presence of confounding bias poses a key challenge in policy evaluation, as the target causal effects of actions are not identifiable (i.e., underdetermined) from observational data. On the other hand, existing confounding-robust evaluation strategies require detailed prior knowledge about the environment or apply only to discrete treatments and outcomes. This paper investigates causal effect evaluation over the continuous domain from confounded observations, while requiring only basic temporal ordering between the treatment and the outcome. We introduce a universal discretization of the exogenous domains that approximates the observational and interventional distributions of any causal model with arbitrary accuracy using a finite number of latent states. Building on this newfound universal approximation property, we develop a novel family of Causal Gaussian process (CGP) models that effectively approximate the observational and interventional distributions of any causal model with confounded observations.

Junzhe Zhang, Jingyuan Chen, Elias Bareinboim

Department of Electrical Engineering and Computer Science, Syracuse University · Department of Computer Science, Columbia University

Causal Effect Estimation Causal Inference with Unobserved Confounding Policy Evaluation Causal Identifiability Causal Inference

Jun 13, 2026

AI for Social Good: An Investigation of the Causal Relationship Between Environmental Regulations and Their Effects on Air Pollution in London, UK

Air pollution regulation is central to urban public health governance, but estimating its effects is difficult because policies are implemented non-randomly and pollution trajectories are shaped by meteorology, socioeconomic change, temporal trends, and overlapping interventions. This study develops an uncertainty-aware Bayesian deep learning framework to estimate the aggregate effect of air pollution regulations on PM

_{2.5}

concentrations in London from 2010 to 2020. The framework integrates daily PM

_{2.5}

observations from Inner London monitoring stations, meteorological covariates, annual socioeconomic indicators, month-of-year and day-of-week indicators, and daily regulation status data for 32 policy measures. A Bayesian LSTM captures temporal dependencies in environmental and socioeconomic covariates, Bayesian embedding layers represent temporal and regulation status inputs, and a regulation status prediction branch supports propensity score-based adjustment for non-random policy implementation. Regulatory effects are estimated by comparing observed PM

_{2.5}

concentrations with counterfactual predictions under a hypothetical no-regulation scenario, with uncertainty summarized across repeated Bayesian training runs and bootstrap resampling. Results show that London's regulations were associated with an average PM

_{2.5}

reduction of 1.88

μ

g/m

^3

, a relative reduction of 12.35%, with a 95% confidence interval of 1.64-2.12

μ

g/m

^3

. Estimated effects were limited before 2013, became clearer from 2013 to 2017, and were strongest in 2018 and 2019. The findings suggest that sustained and cumulative regulatory interventions contributed to measurable improvements in London's air quality. This study demonstrates how uncertainty-aware causal AI can support environmental accountability, public health protection, and evidence-based governance for environmental decision-making.

Yang Han, Jacqueline CK Lam, Victor OK Li +1

Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong

Causal Effect Estimation Policy Evaluation

Jun 7, 2026

Evaluating AI Investment Strategies

We study the problem of auditing a black-box algorithmic decision-maker from observable inputs and outputs alone. Our main result is an exact decomposition: under precisely characterized conditions, the cumulative regret of a dynamic policy equals the sum of per-period covariances between the cost vector and the policy's decision. This extends the single-period identity of Aldridge~(2026) to the full multi-period setting of stochastic dynamic programming. We prove the identity holds exactly under i.i.d. costs and mean-unbiased Markov policies, derive closed-form bias corrections for non-stationary and time-varying cases, and establish the discounted-horizon analog. A Bellman recursion for the covariance regret functional connects the result to standard reinforcement learning algorithms; for rolling-window policies, the estimation-error bias is

O(d/w)

. The decomposition has direct implications for algorithmic auditing in strategic environments: in platform mechanism design, it provides a welfare-based audit metric without access to the agent's private type; in repeated games, covariance reduction is a sufficient condition for policy improvement; in procurement and ad auctions, the bias correction quantifies welfare loss from strategic misreporting. The associated trajectory estimator is consistent, asymptotically normal with HAC variance, and computable in

O(T \cdot nd)

time. This makes the proposed approach a tractable, model-free audit tool for platform mechanisms, algorithmic portfolio strategies, and any sequential decision system subject to external performance review.

Irene Aldridge

Algorithmic Auditing Policy Evaluation Black-Box Model Auditing

Jun 4, 2026

PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation

Vision-language-action (VLA) policies operate in a closed loop in real-world robot tasks: a robot observes the scene, executes an action chunk, and conditions its next decision on the resulting observation. However, most existing world models for robot action evaluation are limited to open-loop prediction along pre-collected action trajectories. This prevents them from supporting closed-loop VLA evaluation, where each action chunk must be conditioned on the observation generated by the previous execution. To address this gap, we propose PiL-World, a chunk-wise world model designed for policy-in-the-loop VLA evaluation. Given the current observation and the action trajectory rolled out by a VLA policy, PiL-World generates multi-view future observations that are consistent with the VLA rollout and match the image inputs required by the policy. By alternating between VLA inference and world-model prediction, PiL-World enables closed-loop evaluation without real robot execution at every step. To improve rollout fidelity, PiL-World conditions video generation on action-derived visual control from head-view robot motion and latent histories that encode task execution context, while jointly predicting complementary multi-view observations. Beyond successful teleoperated demonstrations, it also learns from failed execution trajectories, helping the imagined rollouts better match the distribution of real policy executions. We evaluate PiL-World on three real dual-arm manipulation tasks. PiL-World generates imagined rollouts that are highly consistent with real robot executions. More importantly, compared with the baseline, it reduces the error between VLA success rates measured in real-world rollouts and those estimated through closed-loop world-model evaluation from 63.2% to 12.0%.

Chong Ma, Taiyi Su, Jian Zhu +4

1Tongji University · 2AIRC, Midea Group

Action-Conditioned World Models Robotic Policy Evaluation Policy Evaluation Vision-Language-Action Models World Model Evaluation Embodied World Models+1

Jun 3, 2026

X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation

Rigorous evaluation of learning-based robotic systems is an essential prerequisite for deployment. However, real-world test data is expensive to gather; moreover, in a typical iterative development context, data gathered from the latest policy is necessarily limited in scale. This motivates evaluation methodologies that make use of heterogeneous data sources, including simulation, historical policy logs, and data collected from related platforms or environments. While such auxiliary data are abundant and inexpensive, they are generally not directly representative of real-world outcomes -- for example, performance in simulation may differ substantially from performance in the real world -- making their principled use for high-confidence performance estimation challenging. In this paper, we introduce X4Val, a general framework for variance-reduced real-world metric estimation in the presence of non-paired, multi-domain data. X4Val embeds samples from real and auxiliary domains into a shared representation space and learns a transferable predictor of real-world metrics; this learned predictor is then incorporated into a control-variates estimator, enabling variance reduction even when paired samples are unavailable. We provide theoretical analysis and empirical evaluations on autonomous driving and real-world robot manipulation tasks, domains across which X4Val achieves up to 38.4% variance reduction and demonstrates consistent improvements over strong baselines. These results show that non-paired, heterogeneous data can be leveraged to substantially improve the sample efficiency of rigorous robotic system validation.

Rachel Luo, Michael Watson, Apoorva Sharma +6

NVIDIA Research, Santa Clara, CA, USA · Harvard University, Cambridge, MA, USA · Stanford University, Stanford, CA, USA

Off-Policy Evaluation Robotic Policy Evaluation Policy Evaluation Variance Reduction

May 26, 2026

Margin Play: A Multi-Agent System For Public Policy Analysis In The Brazilian Equatorial Margin

The Brazilian Equatorial Margin (BEM) is Brazil's next offshore oil frontier, with operations expected to begin in 2026 in the Foz do Amazonas basin. Its assets are fiscally and territorially linked primarily to Maranhao -- the state with the lowest HDI in the Federation (0.676, IBGE 2022). This raises the central policy question: under what conditions does BEM exploration generate net positive externalities for Maranhao? The problem is intrinsically multi-agent: the Federal Government seeks revenue and energy security; the state seeks regional welfare under constitutional royalty earmarking; the operator maximizes profit under risk; ANP and IBAMA hold conflicting mandates; and Amazonian communities prioritize territorial and environmental vectors over monetary income. We present Margin Play, a Multi-Agent Reinforcement Learning (MARL) system simulating these tensions under Brazilian empirical calibration and classical economic literature. It implements six agents under the CTDE paradigm, trained with BRO-MARL. Results from 60,000 episodes across six scenarios indicate the answer is conditional on the institutional regime: under the reference baseline, the welfare gain is marginal (Waval approx. 1.68), whereas the MA-Prospero configuration yields Delta W = +17.5% and Delta Rcom = +21.3%, with a lower environmental liability (Eamb = 0.048 vs. 0.076). The fundamental problem is not a trade-off between production and welfare, but the choice of public policy regime linked to exploration.

Antonio de Sousa Leitão Filho, Fabrício Saul Lima, Selby Mykael Lima dos Santos +4

Aia Context, São Luís, MA, Brazil · Universidade Federal do Maranhão — UFMA, São Luís, MA, Brazil · Universidade Estadual de Campinas — UNICAMP, Campinas, SP, Brazil

Policy Evaluation

May 13, 2026

Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readiness

Marketplace platforms routinely evaluate pricing and allocation policies using logged observational data, yet strong offline performance does not imply that a policy is safe to deploy. In real-time bidding (RTB) marketplaces, reserve-price and floor-policy changes affect not only revenue but also fill, advertiser value, budget pacing, and competition across auctions, creating feedback and interference. The central problem is therefore not to estimate whether a policy improves an offline metric, but to determine whether the available evidence justifies direct launch or only further validation. In this regard, we propose a support-aware decision-support system (DSS) that distinguishes promising from actionable evidence. The framework integrates replay, support-aware off-policy evaluation (OPE), conservative lower-bound ranking, multi-sided guardrails, out-of-time validation, sensitivity analysis, and interference-aware validation design into a claim-preserving pipeline that outputs a launch-readiness classification rather than a single performance estimate. Applying the framework to iPinYou-style RTB logs, we identify a margin-gated floor policy as the leading candidate, with a 47.7% replay yield lift, a 45.8% conservative lower-tail lift, and stable out-of-time performance. However, the framework does not recommend direct launch. A decision-rule ablation shows that simplified pipelines select the same policy but incorrectly recommend deployment, leaving key causal assumptions unresolved. In contrast, the proposed DSS selects the same policy but changes the action to online validation, reflecting missing evidence on propensities, bidder response, and interference. Overall, the contribution is a reproducible DSS protocol that prevents decision overclaim under partial identification and converts offline evaluation into an auditable, action-oriented recommendation.

Prashant Shekhar, Caroline Howard

Department of Mathematics Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

AI-Assisted Decision Making Off-Policy Evaluation Online Advertising Policy Evaluation Partial Identification Real-Time Bidding

May 3, 2026

Bridging the Gap Between Average and Discounted TD Learning

The analysis of Temporal Difference (TD) learning in the average-reward setting faces notable theoretical difficulties because the Bellman operator is not contractive with respect to any norm. This complicates standard analyses of stochastic updates that are effective in discounted settings. Although a considerable body of literature addresses these challenges, existing theoretical approaches come with limitations. We introduce a novel algorithm designed explicitly for policy evaluation in the average-reward setting, utilizing sampling from two Markovian trajectories. Our proposed method overcomes previous limitations by guaranteeing convergence to the unique solution of a properly defined projected Bellman equation. Notably, and in contrast to earlier work, our convergence analysis is uniformly applicable to both linear function approximation and tabular settings and does not involve explicit dimension-dependent terms in its convergence bounds. These results align with what is known to hold in the discounted setting. Furthermore, our algorithm achieves improved dependence on the problem's condition number, reducing the sample complexity from quartic, as in prior literature, to quadratic scaling, and thus matching the efficiency seen in the discounted setting.

Haoxing Tian, Zaiwei Chen, Ioannis Ch. Paschalidis +1

Department of Electrical and Computer Engineering Boston University · Edwardson School of Industrial Engineering Purdue University

Reinforcement Learning Average-Reward RL Stochastic Approximation Policy Evaluation Temporal-Difference Learning Linear Function Approximation

Apr 13, 2026

Structural Consequences of Policy-Based Interventions on the Global Supply Chain Network

As global political tensions rise and the anticipation of additional tariffs from the United States on international trade increases, the issues of economic independence and supply chain resilience become more prominent. The importance of supply chain resilience has been further underscored by disruptions caused by the COVID-19 pandemic and the ongoing war in Ukraine. In light of these challenges, ranging from geopolitical instability to product supply uncertainties, governments are increasingly focused on adopting new trade policies. This study explores the impact of several of these policies on the global electric vehicle (EV) supply chain network, with a particular focus on their effects on country clusters and the broader structure of international trade. Specifically, we analyse three key policies: Country Plus One, Friendshoring, and Reshoring. Our findings show that Friendshoring, contrary to expectations, leads to greater globalisation by increasing the number of supply links across friendly countries, potentially raising transaction costs. The Country Plus One policy similarly enhances network density through redundant links, while the Reshoring policy creates challenges in the EV sector due to the high number of irreplaceable products. Additionally, the effects of these policies vary across industries; for instance, mining goods being less affected in Country Plus One than the Friendshoring policy.

Lea Karbevska, Liming Xu, Zehui Dai +2

Department of Engineering, University of Cambridge, Cambridge, United Kingdom · The Alan Turing Institute, London, United Kingdom

Supply Chain Management Policy Evaluation

Mar 11, 2026

LLM-Augmented Digital Twin for Policy Evaluation in Short-Video Platforms

Short-video platforms are closed-loop, human-in-the-loop ecosystems where platform policy, creator incentives, and user behavior co-evolve. This feedback structure makes counterfactual policy evaluation difficult in production, especially for long-horizon and distributional outcomes. The challenge is amplified as platforms deploy AI tools that change what content enters the system, how agents adapt, and how the platform operates. We propose a large language model (LLM)-augmented digital twin for short-video platforms, with a modular four-twin architecture (User, Content, Interaction, Platform) and an event-driven execution layer that supports reproducible experimentation. Platform policies are implemented as pluggable components within the Platform Twin, and LLMs are integrated as optional, schema-constrained decision services (e.g., persona generation, content captioning, campaign planning, trend prediction) that are routed through a unified optimizer. This design enables scalable simulations that preserve closed-loop dynamics while allowing selective LLM adoption, enabling the study of platform policies, including AI-enabled policies, under realistic feedback and constraints.

Haoting Zhang, Yunduan Lin, Jinghai He +3

University of California, Berkeley · The Chinese University of Hong Kong · New York University +1

Policy Evaluation Digital Twins Large Language Model-Based Social Simulation

Feb 26, 2026

Accelerated Online Risk-Averse Policy Evaluation in POMDPs with Theoretical Guarantees and Novel CVaR Bounds

Risk-averse decision-making under uncertainty in partially observable domains is a central challenge in artificial intelligence and is essential for developing reliable autonomous agents. The formal framework for such problems is the partially observable Markov decision process (POMDP), where risk sensitivity is introduced through a risk measure applied to the value function, with Conditional Value-at-Risk (CVaR) being a particularly significant criterion. However, solving POMDPs is computationally intractable in general, and approximate methods rely on computationally expensive simulations of future agent trajectories. This work introduces a theoretical framework for accelerating CVaR value function evaluation in POMDPs with formal performance guarantees. We derive new bounds on the CVaR of a random variable X using an auxiliary random variable Y, under assumptions relating their cumulative distribution and density functions; these bounds yield interpretable concentration inequalities and converge as the distributional discrepancy vanishes. Building on this, we establish upper and lower bounds on the CVaR value function computable from a simplified belief-MDP, accommodating general simplifications of the transition dynamics. We develop estimators for these bounds within a particle-belief MDP framework with probabilistic guarantees, and employ them for acceleration via action elimination: actions whose bounds indicate suboptimality under the simplified model are safely discarded while ensuring consistency with the original POMDP. Empirical evaluation across multiple POMDP domains confirms that the bounds reliably separate safe from dangerous policies while achieving substantial computational speedups under the simplified model.

Yaacov Pariente, Vadim Indelman

Technion - Israel Institute of Technology, Israel

Partially Observable Markov Decision Processes Policy Evaluation Risk-Sensitive RL Conditional Value-at-Risk

Feb 25, 2026

PPCR-IM: A System for Multi-layer DAG-based Public Policy Consequence Reasoning and Social Indicator Mapping

Public policy decisions are typically justified using a narrow set of headline indicators, leaving many downstream social impacts unstructured and difficult to compare across policies. We propose PPCR-IM, a system for multi-layer DAG-based consequence reasoning and social indicator mapping that addresses this gap. Given a policy description and its context, PPCR-IM uses an LLM-driven, layer-wise generator to construct a directed acyclic graph of intermediate consequences, allowing child nodes to have multiple parents to capture joint influences. A mapping module then aligns these nodes to a fixed indicator set and assigns one of three qualitative impact directions: increase, decrease, or ambiguous change. For each policy episode, the system outputs a structured record containing the DAG, indicator mappings, and three evaluation measures: an expected-indicator coverage score, a discovery rate for overlooked but relevant indicators, and a relative focus ratio comparing the systems coverage to that of the government. PPCR-IM is available both as an online demo and as a configurable XLSX-to-JSON batch pipeline.

Zichen Song, Weijia Li

Lanzhou University

Policy Evaluation LLM Reasoning with Graphs

Feb 23, 2026

Denotational Semantics for ODRL: Knowledge-Based Constraint Conflict Detection

ODRL's six set-based operators -- isA, isPartOf, hasPart, isAnyOf, isAllOf, isNoneOf -- depend on external domain knowledge that the W3C specification leaves unspecified. Without it, every cross-dataspace policy comparison defaults to Unknown. We present a denotational semantics that maps each ODRL constraint to the set of knowledge-base concepts satisfying it. Conflict detection reduces to denotation intersection under a three-valued verdict -- Conflict, Compatible, or Unknown -- that is sound under incomplete knowledge. The framework covers all three ODRL composition modes (and, or, xone) and all three semantic domains arising in practice: taxonomic (class subsumption), mereological (part-whole containment), and nominal (identity). For cross-dataspace interoperability, we define order-preserving alignments between knowledge bases and prove two guarantees: conflicts are preserved across different KB standards, and unmapped concepts degrade gracefully to Unknown -- never to false conflicts. A runtime soundness theorem ensures that design-time verdicts hold for all execution contexts. The encoding stays within the decidable EPR fragment of first-order logic. We validate it with 154 benchmarks across six knowledge base families (GeoNames, ISO 3166, W3C DPV, a GDPR-derived taxonomy, BCP 47, and ISO 639-3) and four structural KBs targeting adversarial edge cases. Both the Vampire theorem prover and the Z3 SMT solver agree on all 154 verdicts. A key finding is that exclusive composition (xone) requires strictly stronger KB axioms than conjunction or disjunction: open-world semantics blocks exclusivity even when positive evidence appears to satisfy exactly one branch.

Daham Mustafa, Diego Collarana, Yixin Peng +4

RWTH Aachen University, Germany · Fraunhofer FIT, Sankt Augustin, Germany · University of Galway, Ireland

Policy Evaluation Knowledge Representation Constraint Satisfaction

Feb 23, 2026

Axis-Aligned Semantics for ODRL: Resolving Dimensional Ambiguity in Policy Constraints

The Open Digital Rights Language (ODRL) represents policy constraints as triples of a left operand, an operator, and a value. Several spatial operands, however, range over multi-axis domains such as width, height, and depth, while the constraint syntax provides no explicit axis identity. As a result, policy engines cannot determine whether multiple constraints apply to the same axis or different ones, making conflict detection unsound or incomplete. We resolve this ambiguity by axis decomposition, replacing multi-axis operands with axis-specific scalar operands over totally ordered domains. Each constraint then denotes an interval per axis and each policy an axis-aligned box, reducing conflict detection to box comparison. We define a three-valued semantics (Conflict, Compatible, Unknown), prove the decomposition sound and backward compatible with ODRL, instantiate it as ODRL Axis-Aligned Profile (OAAP), and validate it on a benchmark of 256 ODRL policy problems, each expressed in Turtle and compiled to first-order (TPTP) and SMT-LIB form, using Vampire, E, Z3, and cvc5.

Daham Mustafa, Diego Collarana, Sabrina Kirrane +5

RWTH Aachen University, Aachen, Germany · Fraunhofer FIT, Sankt Augustin, Germany · Vienna University of Economics and Business (WU), Vienna, Austria +1

Policy Evaluation Access Control

Feb 17, 2026

Decision Quality Evaluation Framework at Pinterest

Online platforms require robust systems to enforce content safety policies at scale. A critical component of these systems is the ability to evaluate the quality of moderation decisions made by both human agents and Large Language Models (LLMs). However, this evaluation is challenging due to the inherent trade-offs between cost, scale, and trustworthiness, along with the complexity of evolving policies. To address this, we present a comprehensive Decision Quality Evaluation Framework developed and deployed at Pinterest. The framework is centered on a high-trust Golden Set (GDS) curated by subject matter experts (SMEs), which serves as a ground truth benchmark. We introduce an automated intelligent sampling pipeline that uses propensity scores to efficiently expand dataset coverage. We demonstrate the framework's practical application in several key areas: benchmarking the cost-performance trade-offs of various LLM agents, establishing a rigorous methodology for data-driven prompt optimization, managing complex policy evolution, and ensuring the integrity of policy content prevalence metrics via continuous validation. The framework enables a shift from subjective assessments to a data-driven and quantitative practice for managing content safety systems.

Yuqi Tian, Robert Paine, Attila Dobi +3

Pinterest, San Francisco, CA, USA · Pinterest, New York, NY, USA · Pinterest, Richmond, VA, USA +1

LLM Safety Evaluation Content Moderation LLM Agent Evaluation Policy Evaluation

Nov 24, 2025

Neural Evidence and Behavioural Welfare Economics: When Can Neuroeconomics Inform Policy?

Neuroeconomics appears to offer welfare economics an evidential supplement, since neural data may help distinguish choices that express genuine preferences from choices distorted by addiction, framing or self-control failure. This paper asks when such evidence can legitimately inform welfare judgements for policy. It develops a conceptual framework linking three levels of analysis, namely neural signals, computational decision models and normative welfare criteria. The central argument is that neural data can constrain welfare analysis only under four conditions, covering the validity of the neural measure, the identifiability of the decision model, the explicit statement of the welfare criterion, and the institutional context of any policy use. Applying this framework to addiction, neuromarketing and environmental policy, and working through the United Kingdom's Tobacco and Vapes Act 2026, a major intervention justified without any neural measurement, the paper develops a Neuroeconomic Welfare Inference Checklist for neural evidence. Neural data can improve behavioural welfare economics by clarifying the mechanisms behind choice, and they cannot replace normative argument about what counts as welfare. The same reasoning applies to artificial agents trained on human feedback, whose internal reward signals are computational quantities and not welfare measures.

Louis Yiven Zhu

Student (BSc, 2023-26), Dept. of Science and Technology Studies, UCL, UK

Policy Evaluation

Oct 21, 2024

Statistical Inference for Policy Evaluation with Temporal Difference Learning

We investigate the statistical properties of Temporal Difference (TD) learning with Polyak-Ruppert averaging, arguably one of the most widely used algorithms in reinforcement learning, for the task of estimating the parameters of the optimal linear approximation to the value function. Assuming independent samples, we make three theoretical contributions that improve upon the current state-of-the-art results: (i) we establish refined high-dimensional Berry-Esseen bounds over the class of convex sets, achieving faster rates than the best known results, and (ii) we propose and analyze a novel, computationally efficient online plug-in estimator of the asymptotic covariance matrix; (iii) we derive sharper high probability convergence guarantees that depend explicitly on the asymptotic variance and hold under weaker conditions than those adopted in the literature. These results enable the construction of confidence regions and simultaneous confidence intervals for the linear parameters of the value function approximation, with guaranteed finite-sample coverage. We demonstrate the applicability of our theoretical findings through numerical experiments.

Weichen Wu, Gen Li, Yuting Wei +1

The Voleon Group, New York, NY, USA · Department of Statistics, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. +1

Parameter Estimation Confidence Region Estimation Value Function Estimation Stochastic Approximation Policy Evaluation Temporal-Difference Learning+3