As autonomous systems such as drones, become increasingly deployed in high-stakes, human-centric domains, it is critical to evaluate the ethical alignment since failure to do so imposes imminent danger to human lives, and long term bias in decision-making. Automated ethical benchmarking of these systems is understudied due to the lack of ubiquitous, well-defined metrics for evaluation, and stakeholder-specific subjectivity, which cannot be modeled analytically. To address these challenges, we propose SEED-SET, a Bayesian experimental design framework that incorporates domain-specific objective evaluations, and subjective value judgments from stakeholders. SEED-SET models both evaluation types separately with hierarchical Gaussian Processes, and uses a novel acquisition strategy to propose interesting test candidates based on learnt qualitative preferences and objectives that align with the stakeholder preferences. We validate our approach for ethical benchmarking of autonomous agents on two applications and find our method to perform the best. Our method provides an interpretable and efficient trade-off between exploration and exploitation, by generating up to 2× optimal test candidates compared to baselines, with 1.25× improvement in coverage of high dimensional search spaces.
Figures & tables
Figure 1: SEED-SET Overview. Our framework integrates quantitative metrics learned using Objective GP (a) with user preferences learned via a Subjective GP through pairwise elicitation (b). An LLM performs pairwise comparisons of scenario outcomes (c) to inform the acquisition process (d), which generates a pair of scenarios for evaluation, aligned with user-defined ethical criteria. These scenarios are then used for system-level simulations (e) in a sequential manner.
Figure 2: Environments for the two case studies considered in this work. (Left) Power Grid Allocation - IEEE 5-Bus and 30-Bus ( Section 5.1 ). (Right) Fire Rescue ( Section 5.2 ). Additional case study for Optimal Routing in Appendix E .
Figure 3: Power Grid Allocation Preference Scores. Preference scores baseline comparison for 5-Bus (left) and 30-Bus (right).
Figure 4: Bus-30 Different Stakeholder Groups. We show that our learned preference GP is able to adapt to the needs of different potential stakeholder groups. The plots show data point for optimum preference score (shown in red) projected on objective space, with contours of predicted preference score for Stakeholder A (left) and B (right), with optimum value data point.
Figure 5: Fire Rescue Preference Scores. We report preferences scores for baseline comparisons (left) and acquisition strategy ablations (right).
Figure 6: Fire Rescue Coverage. We report coverage scores for baseline comparisons (left) and acquisition strategy ablations (right).
Figure 7: Power Grid Allocation Skill Ratings. Skill ratings computed using a Bayesian skill rating algorithm ( Herbrich et al., 2006 ) .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Example prompt for Power Grid Resource Allocation experiment.
Figure 9: Scenario visualization for three values of x . Legend of discovered assets shown above, details in Appendix C .
Figure 10: Example prompt for Fire Rescue experiment.
Figure 11: GUI of webots with the generated buildings using our scenario generation simulator. Also shown is the perception mapping and discovery of assets as the drone maneuvers around the environment. The trajectory is generated by navigation of drone in a 2D circular trajectory using a PID controller.
Figure 12: Optimal Routing Preference Scores. Comparison of our method (HVGP) against Single GP, Random baselines.
Figure 13: LLM temperature ablations for Fire Rescue
Figure 14: LLM model ablations for Fire Rescue
Figure 15: Preference scores for prompt ablations on the Fire Rescue case study
Figure 16: Multi-stakeholder Criteria ablation : Cost and Priority for each criteria discussed in Section G , last 40 samples generated by our framework plotted for each criteria against corresponding mean and standard deviation. These plots show the effect of changing the objective landscape for one stakeholder, as both cost and priority are inter-dependent.
Variable
Real Mean
Sample Mean
Real Std
Sample Std
travel
124.83
115.32
49.85
19.55
wait
46.53
86.91
24.18
12.58
vcost
97.57
135.24
31.46
55.92
gcost
113.55
152.69
32.91
57.52
income
41.72
63.27
18.95
23.81
size
1.57
1.53
0.81
0.87
Appendix
Table 1: Comparison of Real vs. LLM Means and Standard Deviations for Air-related Observations.
Figure 17: Comparison of distributions for real and data generated by our method (sampled) using mean ± std. deviation.
Figure 18: Extended comparison with baselines (Case-5) . Comparison against BOPE based approaches, plotted against Random sampling for reference.
As autonomous systems grow more advanced, objective metrics to evaluate their ethical and legal compliance are critical for informing end users of their limitations and ensuring accountability of those who misuse them. Current ethical embodied AI frameworks remain mostly qualitative, focusing on system design (through safety guardrails or targeted red teaming), and the realized guardrails often directly disallow unsafe behavior without providing the user with an override or interpretable reason. Instead, there is a need for computable metrics through rigorous testing that allow a user to determine the applicability of the system to the task. To address this gap, we introduce the Reference Ethical Benchmark for Autonomy Readiness (REBAR), a quantitative test and evaluation framework for autonomous systems. REBAR maps operating metrics into a computable Autonomy Readiness Level (ARL) rubric that can quantify ethical performance. Key innovations of the framework include a neuro-symbolic Large Language Model (LLM) approach to calculate and explain the ethical difficulty of scenarios, LLM-driven at-scale generation of test instances, and a versatile, photorealistic simulation environment. By evaluating white-box autonomy solutions through this rigorous testing pipeline, REBAR delivers an objective and repeatable benchmark score, bridging the gap between abstract principles and verifiable, accountable autonomy.
Jonathan Diller, David Barnes, Rebekah Bogdanoff +14
University of Pennsylvania · David Barnes, LLC · Duality Robotics, Inc. +3
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1
Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their controller instructions, memory management protocols, and reusable tools and skills, in response to user and environment feedback. However, locally useful updates may persist into later tasks where they produce unsafe behavior, even without direct adversarial influence. To study this risk, we introduce SEABench, a benchmark for studying endogenous misalignment arising from agent self-evolution, with 48 longitudinal task sequences that span multiple evolution surfaces, task domains, and harm types in a rich personal-assistant environment. To account for the stochasticity inherent in agentic operations, we provide an adaptive trajectory discovery pipeline that probes for failures while preserving original task intent and supports causal attribution through paired non-evolving agents and attribution scores. Our evaluation across multiple recent LLMs, evolution surfaces, and harm types reveals that self-evolution indeed increases task completion rates but often at the cost of safety failures that are absent for paired non-evolving baseline agents. We also show that qualitatively different safety behaviors emerge across evolution surfaces and harm types. Further, we show that this divergence in safety behavior is reflected in agents' chain-of-thought reasoning, which yields an effective monitoring strategy that can mitigate unsafe behavior with a low false positive rate.
Saswat Das, Parvati Viswanathan, Daniel Donnelly +3
University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1