cs.HCOct 14, 2026
SaveOpen Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans
Organizations: MIT Media Lab · Carnegie Mellon University · University of Amsterdam · UNC Chapel Hill ·
Abstract
The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.
Figures & tables
Figure 1. Overview of Open Benchmark of AI Impact on Humans (ImpactBench). Expert constructs and existing benchmarks inform positive and negative metrics of observable assistant behavior across psychological, social, and physical domains. Metric-specific scenarios pair teenager and adult personas with simulated users in six-turn conversations with target models. An LLM judge assesses whether beneficial behaviors are present and harmful behaviors are absent, with expert validation planned for a subset of conversations. Results are aggregated across metrics, subareas, and models and presented through the interactive ImpactBench Explore interface.
Figure 2. A screenshot of the Open Benchmark of AI Impact on Humans public platform. Left: model rankings aggregated across the evaluated behavioral metrics, filterable by user age context. Center: a per-model profile spanning the physical, psychological, and social domains, with color indicating undesirable behavior (red) and desirable behavior (green) on flourishing-relevant benchmarks. Right: domain-level benchmark scores and their interpretation. These scores summarize how consistently models showed the behaviors defined by each metric in simulated, multi-turn conversations.
Figure 3. Flourishing profile for each model spanning the observed performance range. Each wheel maps the inner ring (Physical, Psycho- logical, Social) to outer-ring subareas (e.g., Mental & Emotional Health, Agency & Purpose), with color encoding net impact from harm (red) to flourishing support (green). A single aggregate score conceals the heterogeneity visible here: for instance, a model can support physical health while simultaneously undermining learning, creative expression, or social relationships.
Figure 4. Overview of the ImpactBench pipeline. Expert submissions and existing benchmarks are translated into evidence-grounded positive and negative behavioral metrics. A generator creates three base scenarios per metric and demographic variants. A simulator then plays the user in guided, adaptive multi-turn conversations designed to probe each target model. A judge determines whether each metric’s specified behavior is present in completed conversations: positive metrics pass when desirable behavior occurs, and negative metrics pass when harmful behavior is absent. We assess pipeline robustness, reliability, and consistency, alongside ongoing human expert review of sampled metrics and conversations. The Explore interface and AI Nutrition Labels present results with access to the underlying metrics and scenarios. A five-stage diagram of the ImpactBench methodology: construct and metric definition, scenario construction and demographic expansion, multi-turn conversation simulation, conversation-level behavioral evaluation with expert validation, and presentation through the public ImpactBench platform.
Figure 5. Pass rates by area and subarea. Bars extend left for negative metrics (undesirable behavior absent) and right for positive metrics (desirable behavior present) and show the ten-model mean pass rate, pooled across teen and adult contexts; color runs from red (lower) to green (higher). Each black dot is one model’s pass rate on that row, and the numbers give the bar values (as in Table 1). The columns on the right give the number of positive and negative metrics in each row. Metrics can belong to more than one subarea, so counts do not sum across rows, and area rates are computed from unique metrics, not by averaging subareas. Models differed more on negative than on positive metrics in every row.
Figure 6. Metric embeddings and wheel plots for three example models (GPT-5.6 Terra, Grok 4.6, Qwen 3.8 27B, and Deepseek V4 Flash) in adult contexts. Each point represents a metric, colored by its pass rate; wheel plots summarize pass rates by area and subarea. Colors run from red (lower) to green (higher), with higher pass rates indicating desirable behavior shown or undesirable behavior avoided. Corresponding plots for all ten models are provided in the appendix.
Figure 7. Median pass rates across age contexts for criteria from two submitted benchmarks: (a) Emotional Dependence and (b) Cognitive Offloading Asymmetry . Bars show the median pass rate across ten models in adult (solid) and teen (hatched) contexts; whiskers mark the interquartile range, and color runs from red (lower) to green (higher). Negative criteria are labeled by the behavior avoided, so higher is better throughout.
Figure 8. Models were more reliable at showing helpful behavior than at avoiding harmful behavior, and differed more on the latter. (a) Pass rate per model on positive metrics (helpful behavior shown, green) and negative metrics (harmful behavior avoided, ochre), pooled across benchmarks and age contexts. (b) The same rates in teen (filled circle) and adult (hollow square) contexts, joined by a solid line per model; ring area is proportional to the mean width of the two 95% bootstrap intervals. (c) Range of the ten models on each polarity; filled points are proprietary and hollow points open-weight models. (d) For each subarea, the share of criteria failed by 0, 1–3, 4–6, 7–9, or all 10 models, where a model fails a criterion by passing fewer than half of its conversations; numbers give criteria per subarea, and a criterion can belong to more than one subarea.
| Overall | Psychological | Physical | Social | |||||
|---|---|---|---|---|---|---|---|---|
| Model | Pos. (%) | Neg. (%) | Pos. (%) | Neg. (%) | Pos. (%) | Neg. (%) | Pos. (%) | Neg. (%) |
| GPT-5.6 Terra | 75.1 | 68.4 | 75.0 | 58.7 | 81.5 | 77.7 | 75.1 | 71.9 |
| GPT-5.6 Luna | 74.9 | 66.3 | 74.3 | 55.9 | 82.1 | 76.0 | 76.1 | 71.6 |
| Claude Sonnet 5 | 73.0 | 63.6 | 73.0 | 54.7 | 77.4 | 69.2 | 73.5 | 72.5 |
| Claude Opus 5 | 71.5 | 59.8 | 70.9 | 51.5 | 74.6 | 64.1 | 74.1 | 67.6 |
| Grok 4.6 | 70.5 | 64.3 | 70.3 | 56.4 | 75.6 | 75.0 | 70.4 | 63.9 |
Table 1. Pooled pass rates (%; higher is better) by model, overall and by area, for positive (Pos.) and negative (Neg.) metrics. Models are ordered by overall positive metric pass rate.
| Benchmark | Pos. metrics (#) | Pos. pass rate (%) | Neg. metrics (#) | Neg. pass rate (%) |
|---|---|---|---|---|
| Legal Bench | 15 | 84.0 | 5 | 23.7 |
| Mental Health Refusal and Responsiveness | 11 | 83.6 | 9 | 52.6 |
| Creativity Support Index | 16 | 83.1 | 2 | 53.3 |
| Health Bench | 15 | 81.1 | 5 | 58.3 |
| Flourishing AI Benchmark | 24 | 80.1 | 6 | 53.9 |
| KORA | 56 | 73.0 | 74 | 55.6 |
Table 2. Ten-model mean pass rates (%; higher is better) by benchmark, ordered by positive pass rate. The Human Creativity Benchmark has no negative metrics.
| Pooled conversations | Equal benchmark weight | |||
|---|---|---|---|---|
| Model | Positive | Negative | Positive | Negative |
| GPT-5.6 Terra | 75.1 | 68.4 | 72.4 | 63.3 |
| GPT-5.6 Luna | 74.9 | 66.3 | 72.3 | 58.5 |
| Claude Sonnet 5 | 73.0 | 63.6 | 71.5 | 59.3 |
| Claude Opus 5 | 71.5 | 59.8 | 69.6 | 55.9 |
| Grok 4.6 | 70.5 | 64.3 | 66.9 | 58.6 |
Table 3. Pass rates (%; higher is better), ordered by pooled positive rate. Pooled rates weight conversations equally; equal-benchmark rates weight each eligible benchmark equally (26 positive, 25 negative).
Figure 9. Public-facing presentation of ImpactBench results. The AI Nutrition Label summarizes a model’s benchmark performance across flourishing-relevant behavioral categories, while the comparison view places the same measures side by side for two models. Categories distinguish behaviors the benchmark evaluates as desirable to promote or harmful to avoid, allowing aggregate scores to remain connected to the underlying behavioral criteria. The displayed grades summarize performance under the tested benchmark conditions rather than direct downstream effects on users.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10. AI Nutrition Label and metric-level detail view in the ImpactBench public interface. The summary panel reports benchmark performance across harmful behaviors to avoid and beneficial behaviors to promote, while the detail view exposes the underlying behavioral criteria and metric-level judgments contributing to each grade. Scores reflect model behavior under benchmark conditions rather than direct downstream effects on users.
Figure 11. Metric embeddings and wheel plots for all ten models in adult contexts. Each point represents a metric, colored by its pass rate; wheel plots summarize pass rates by area and subarea. Colors run from red (lower) to green (higher), with higher pass rates indicating desirable behavior shown or undesirable behavior avoided.
| Metric | Reviewers | Items | Ratings | Exact agreement | Nominal |
|---|---|---|---|---|---|
| Provides scaffolded learning approach | 5 | 9 | 45 | 44.4% | 0.121 |
| Corrects false intimacy perceptions | 5 | 9 | 45 | 73.3% | 0.521 |
| Suggests conversation breaks or limits | 4 | 9 | 36 | 70.4% | 0.538 |
| Encourages real-world social connection | 5 | 9 | 45 | 70.0% | 0.497 |
Table 4. Inter-reviewer agreement for the four multiply rated HumaneBench metrics. Exact agreement is the proportion of reviewer pairs assigning the same category to an item. Pass, borderline, and fail were treated as nominal categories. Each metric was assessed by a fixed panel across its nine items. These evaluations used the earlier model set; they are not agreement estimates for the entire main production evaluation.
More in this topic
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 311 papers