Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark

Yao Wang; Xin Liu; Zhuochen Liu; Jiankang Chen; Adam Jatowt; Kyoungsook Kim; Noriko Kando; Haitao Yu

cs.CLMar 18, 2026

Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark

Authors: Yao Wang, Xin Liu, Zhuochen Liu, Jiankang Chen, Adam Jatowt, Kyoungsook Kim, Noriko Kando, Haitao Yu

Organizations: Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Japan · Intelligent Platforms Research Institute, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan · Digital Science Center, University of Innsbruck, Innsbruck, Austria · National Institute of Informatics, Tokyo, Japan ·

Abstract

Existing human value datasets do not directly support value understanding in factual news: many are actor-agnostic, rely on isolated utterances or synthetic scenarios, and lack explicit event structure or value direction. We present NEVU (News Event-centric Value Understanding), a benchmark for actor-conditioned, event-centric, and direction-aware human value recognition in factual news. NEVU evaluates whether models can identify value cues, attribute them to the correct actor, and determine value direction from grounded evidence. Built from 2{,}865 English news articles, NEVU organizes annotations at four semantic unit levels (Subevent, behavior-based composite event, story-based composite event, and Article) and labels \mbox{(unit, actor)} pairs for fine-grained evaluation across local and composite contexts. The annotations are produced through an LLM-assisted pipeline with staged verification and targeted human auditing. Using a hierarchical value space with 54 fine-grained values and 20 coarse-grained categories, NEVU covers 45{,}793 unit--actor pairs and 168{,}061 directed value instances. We provide unified baselines for proprietary and open-source LLMs, and find that lightweight adaptation (LoRA) consistently improves open-source models, showing that although NEVU is designed primarily as a benchmark, it also supports supervised adaptation beyond prompting-only evaluation. Data availability is described in Appendix~\ref{app:data_code_availability}.

Figures & tables

Figure 1: Human Value Understanding in News: Core Components and Challenges.

Figure 2: Evidence-Grounded Value Understanding for News-Domain Texts.

Figure 3: Phase-1 pipeline for corpus construction.

Figure 4: An illustrative example of the joint construction of the event-centric knowledge base.

Figure 5: Phase-3 human value annotation pipeline for NEVU.

Figure 6: Illustrative examples of actor-conditioned human value annotations across article, subevent, BCE, and SCE levels.

Figure 7: Level-wise annotation statistics on the full dataset (

s=\texttt{all}

; train+dev+test): (a) numbers of units, annotated unit–actor pairs, and directed value instances; (b) actor density per unit; (c) directed value labels per unit–actor pair; (d) aligned vs. contradictory label ratios across semantic levels.

Group	Models
G1: Proprietary API	OpenAI gpt-5.2 [ 90 ] , gpt-4.1 [ 91 ] , gpt-o3 [ 92 ] ; Claude Haiku [ 93 ] /Sonnet [ 94 ] /Opus 4.5 [ 95 ] ; Gemini-2.5-Pro [ 96 ] ; Deepseek-chat [ 97 ] , Deepseek-reasoner [ 98 ]
G2: Open-weight (w/o fine-tuning)	Llama-3.1-8B-Instruct [ 99 ] , Qwen3-8B [ 100 ] , Ministral-3-8B-Instruct-2512 [ 101 ] , Phi-3.5-mini-instruct [ 102 ]
G3: Open-weight (fine-tuned)	Llama-3.1-8B-Instruct + LoRA, Qwen3-8B + LoRA, Ministral-3-8B-Instruct-2512 + LoRA, Phi-3.5-mini-instruct + LoRA

Table 3: Model groups used in the Human Value Recognition benchmark. G1 and G2 are evaluated in a prompting-only setting, while G3 applies LoRA fine-tuning to the same open-weight backbones. All groups use a shared prompt template and actor-centric JSON output schema.

Group	Model	Micro-F1					Macro-F1					DRR $\downarrow$
Group	Model	Art.	Sub.	BCE	SCE	Overall	Art.	Sub.	BCE	SCE	Overall	Art.	Sub.	BCE	SCE	Overall
G1	Claude Haiku 4.5	43.05	41.02	38.38	39.69	40.87	28.87	33.13	28.18	32.93	30.11	3.44	1.72	2.41	2.43	2.74
	Claude Sonnet 4.5	46.37	44.91	42.17	43.22	44.43	32.54	39.03	35.41	36.37	34.73	2.61	1.22	1.97	2.19	2.15
	Claude Opus 4.5	47.76	46.68	44.47	45.68	46.37	34.86	41.00	37.07	37.33	37.25	2.82	1.00	2.00	1.92	2.16
	Deepseek-chat	36.62	39.76	39.04	40.98	38.67	23.05	30.32	29.95	30.23	26.82	2.64	1.36	2.31	2.53	2.32
	Deepseek-reasoner	43.47	43.45	43.87	45.23	43.94	31.76	36.81	34.06	37.73	34.92	1.65	1.40	1.42	1.16	1.47

Table 4: Human Value Recognition results on 54 fine-grained Level-1 human values. G1: proprietary API LLMs (prompting only); G2: open-weight LLMs (prompting only); G3: open-weight LLMs with LoRA fine-tuning. Best-in-group is underlined; overall best is underlined in bold.

Set.	Lvl.	Units	(u,a)	Inst.	Align.	Contra.	Aligned ratio
HCP	Art.	192	3513	753	2634	879	75.0
HCP	BCE	650	1991	955	1596	395	80.2
HCP	SCE	569	1979	967	1585	394	80.1
HCP	Sub.	1878	4152	2347	3156	996	76.0
CGT	BCE	412	509	1078	854	224	79.2
CGT	SCE	353	462	905	718	187	79.3

Table 5: Subset statistics (Level-1) for the test-sampled data used in our two diagnostic analyses: hierarchical consistency profiling (HCP) and controlled grouping test (CGT).

	Task (%): Micro-F1 / Micro-P / Micro-R															Consistency (%): mean-F1 / mean-P / mean-R
Model	Art.			Sub.			BCE			SCE			Overall			Art.			BCE			SCE			Overall			$\Delta$ Overall (to Gold)
	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	(pp)
Gold	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	86.5	80.7	96.5	51.5	61.2	48.5	56.2	89.1	46.4	63.0	76.8	61.3	0.0
Llama-3.1-8B	57.6	57.8	57.4	46.8	47.9	45.8	50.8	50.7	50.8	51.9	52.0	51.8	51.7	52.1	51.2	55.3	52.1	66.9	46.0	52.6	44.6	51.9	78.7	43.6	50.7	61.9	50.5	12.3
Ministral-3-8B	57.6	54.6	61.1	47.1	43.2	51.7	49.0	45.7	52.7	52.1	48.8	55.7	51.4	47.9	55.4	56.7	55.4	64.4	41.4	49.2	38.8	48.2	77.0	39.4	48.1	61.0	46.2	14.9
Phi-3.5-mini	53.3	49.2	58.2	46.3	42.1	51.4	49.7	45.6	54.8	52.0	47.3	57.8	50.0	45.7	55.1	55.9	52.3	67.3	49.1	55.4	47.0	55.2	80.9	46.4	53.2	63.7	52.5	9.8

Table 6: Hierarchical Consistency Profiling on Level-1 (54 values). Left: task scores; Right: cross-level consistency scores.

\Delta

Overall denotes the pp gap to the gold Overall consistency.

Figure 8: Controlled Grouping Test (Level-1): Precision/Recall/F1 gains of True composites over Random composites (

\Delta=\textsc{True}-\textsc{Random}

) for SCE and BCE. The Random condition matches the same actor and the same group size

k

by sampling and concatenating

k

actor-associated subevents.

Figure 9: Eval4 news-type comparison (Level-1): Heatmaps show micro-F1 (%) across hierarchical units and news types, while the right bars quantify genre sensitivity using Range (max–min) and Std over the three genres.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Full hierarchy of the human value taxonomy adopted in this work, reproduced from [ 20 ] . The taxonomy consists of 54 Level-1 values grouped into 20 Level-2 categories, aligned with four Level-3 motivational domains, and characterized by two orthogonal Level-4 dimensions: focus (Level-4a: personal / social) and anxiety regulation (Level-4b: growth/anxiety-free / self-protection/anxiety-avoidance).

Source domains
bbc.com	theguardian.com	nytimes.com	eff.org	greenmatters.com
lifehack.org	grist.org	psychologytoday.com	frugalwoods.com	bbc.co.uk
lovepanky.com	pickthebrain.com	securityweek.com	krebsonsecurity.com	markmanson.net
affordanything.com	tinybuddha.com	getrichslowly.org	kasperskycontenthub.com	brenebrown.com
sustainability-times.com	earth911.com	mindful.org	jamesclear.com	konmari.com
positivityblog.com	cupofjo.com	goodfinancialcents.com	schneier.com	thesimpledollar.com

Appendix

Table 7: External source domains referenced by Wikinews pages in our collection (canonicalized domains).

Figure 11: Frequency distribution of the 54 Level-1 human values across Article, Subevent, BCE, and SCE. For each value label, bars compare its count at the four semantic levels; labels are ordered by overall frequency.

Panel of Figure 12 — Figure 12: Screenshots of our manual annotation interface for a BCE human-value labeling example.

Dimension	Subevent	BCE	SCE	Article	Overall
Subevent Evidence Support	91.5/91.0	–	–	–	–
Aggregation Justification	–	95.7/91.4	95.7/91.4	–	–
Evidence Inheritance	–	92.1/84.3	96.4/92.9	–	–
Narrative Coherence	–	–	95.7/91.4	–	–
Local Adjacency Coherence	–	93.6/87.1	–	–	–
Actor Concept Validity	92.5/90.0	97.1/97.1	90.7/88.6	98.9/97.8	94.8/93.4

Appendix

Table 8: Manual quality evaluation results (Soft/Hard, %) with N-weighted overall scores.

Group	Model	Micro-F1					Macro-F1					DRR $\downarrow$
Group	Model	Art.	Sub.	BCE	SCE	Overall	Art.	Sub.	BCE	SCE	Overall	Art.	Sub.	BCE	SCE	Overall
G1	Claude Haiku 4.5	51.18	50.30	46.76	48.62	49.42	38.18	43.01	36.36	40.82	39.71	4.39	2.10	3.28	3.09	3.46
	Claude Sonnet 4.5	53.87	53.48	50.85	51.49	52.56	40.71	47.22	40.28	42.20	42.73	3.24	1.63	2.98	3.05	2.84
	Claude Opus 4.5	55.77	55.53	53.20	54.52	54.86	45.31	50.13	43.34	44.85	46.35	2.88	1.47	2.68	2.63	2.52
	Deepseek-chat	44.27	48.46	47.74	50.22	47.15	31.57	39.41	35.83	37.90	35.61	3.15	1.98	3.06	3.16	2.91
	Deepseek-reasoner	50.45	50.81	51.21	52.41	51.12	40.25	43.55	39.84	44.37	42.26	2.08	1.94	2.19	1.83	2.02

Appendix

Table 9: Human Value Recognition results on 20 coarse-grained Level-2 human values. G1: proprietary API LLMs; G2: open-weight LLMs (prompting only); G3: open-weight LLMs with LoRA fine-tuning. Best-in-group is underlined; overall best is underlined in bold.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum
Jan 20, 2026Víctor Yeste, Paolo RossoMulti-Label Classification Language Model Ensembles
PRHLT Research Center, Universitat Politècnica de València, Valencia, 46022, Spain · School of Science, Engineering and Design, Universidad Europea de Valencia, Valencia, 46010, Spain · Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI)
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
Oct 23, 2026Matt Wilde, Jillian Fisher, Andrew B. Hall +2Generative AI Evaluation Factual Consistency Evaluation
Forum AI · University of Washington · Stanford University
Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement
Sep 22, 2026Yuhe Wu, Rui Qian, Guangyu Wang +10Social Media Analysis
HKUST(GZ) · FDU · DUFE +7

Jan 20, 2026J/K move · Enter open · S save

Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum

Víctor Yeste, Paolo Rosso

PRHLT Research Center, Universitat Politècnica de València, Valencia, 46022, Spain · School of Science, Engineering and Design, Universidad Europea de Valencia, Valencia, 46010, Spain · Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI)

Multi-Label Classification Language Model Ensembles

Open paper AskPDF arXiv

We study neural multi-label classification under severe label imbalance through sentence-level detection of the 19 refined Schwartz human values in 74k English news and manifesto sentences (ValueEval'24 corpus). Each sentence carries a roughly balanced moral-presence label and a 19-way value annotation. First, moral presence is learnable from single sentences: a DeBERTa-base classifier reaches positive-class

F_1 \approx 0.73

at the default threshold, which calibration does not improve. Second, comparing direct multi-label detectors with presence-gated hierarchies under an 8 GB consumer-grade GPU budget, we find that gating does not improve over direct prediction, as gate recall becomes a bottleneck. Third, studying lightweight auxiliary signals and small ensembles, we isolate decision-threshold calibration as a decisive, often overlooked factor: a standard text-only baseline already matches the best official ValueEval'24 English run at the default threshold (macro-

F_1 = 0.282

vs.

\approx 0.28

), and tuning the threshold on validation alone raises it to

0.315

, most of our overall gain. Lightweight features do not survive a paired per-seed test; a soft-voting ensemble reaches our best macro-

F_1 = 0.332

. Calibration is not architecture-specific: it reproduces on RoBERTa-base, where the gain is larger (

+0.043

). To our knowledge, this is the first systematic comparison of direct and presence-gated architectures, lightweight feature-augmented encoders, and instruction-tuned Large Language Models (LLMs) at sentence level; benchmarked 7-9B LLMs (zero-/few-shot and QLoRA) lag behind the supervised ensemble under the same budget. We provide empirical guidance for compute-efficient, value-aware NLP models.

Abstract

LLM Evaluation

Explore similar work

Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum

NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement