Beyond Factual Accuracy: A Benchmark for Evaluating Perspective Alignment in LLMs

Sunghoon Kim; Sangyeop Kim

cs.CLOct 18, 2026

Beyond Factual Accuracy: A Benchmark for Evaluating Perspective Alignment in LLMs

Authors: Sunghoon Kim, Sangyeop Kim

Organizations: Coxwave Seoul, Republic of Korea · Ewha Womans University Seoul, Republic of Korea

Abstract

Large Language Models often generate factually correct responses that fail to maintain user-requested perspectives, blending multiple perspectives and forcing users to filter through unwanted information---a perspective control failure that factual accuracy metrics cannot detect. We introduce PerspectiveBench, a benchmark quantifying perspective alignment by constructing scenarios where multiple valid reasoning paths coexist, and propose the Perspective Alignment Score (PAS) to measure adherence to specified perspectives. Experiments reveal that state-of-the-art models successfully include target content but fail to exclude non-requested perspectives. Providing documents through RAG improves factual grounding but does not resolve perspective drift, as models prioritize comprehensive coverage over strict user constraints. Code and data are available at https://github.com/sunghoon014/PerspectiveBench.

Figures & tables

Figure 1: Perspective misalignment problem.

Figure 2: The PerspectiveBench data construction pipeline: From atomic fact extraction and semantic clustering to graph-based reasoning path and perspective-targeted QA generation.

Domain	Hop	Avg. Total	Avg. Target	Avg. Distractor	Num. Qs
Physics	1	2.77	1.06	1.71	114
	2	6.20	2.60	3.60	161
	3	7.47	3.60	3.86	58
Biology	1	3.34	1.23	2.11	103
	2	5.50	2.35	3.15	164
	3	7.25	2.92	4.33	61

Table 1: Benchmark statistics by domain and hop.

Figure 3: Semantic similarity (bge-m3). Target and distractor paths overlap.

Model	Setting	1-hop					2-hop					3-hop					Overall
		$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok
GPT-5	Base	44.37	92.50	56.13	33.06	464.92	37.33	58.66	42.13	35.56	489.89	35.90	46.79	37.27	37.26	524.71	39.58	68.63	46.26	34.96	486.99
GPT-5	Oracle	48.75	97.50	61.55	68.48	206.17	49.00	71.33	54.41	65.84	241.76	48.07	60.89	49.74	70.23	288.52	48.75	78.82	56.14	67.53	237.13
GPT-4.1	Base	51.50	96.67	63.80	51.88	301.58	39.33	44.00	38.29	48.16	331.95	31.41	39.74	32.75	51.92	361.98	42.29	61.94	46.39	50.12	326.32
GPT-4.1	Oracle	50.16	97.50	62.68	66.64	270.59	39.00	56.00	43.24	62.24	286.85	46.31	65.06	48.93	68.20	351.52	44.21	72.26	51.10	64.82	292.14
Claude-Sonnet-4.5	Base	43.87	90.83	56.12	44.67	405.14	35.00	49.33	37.06	42.10	413.23	34.62	44.87	37.19	44.71	433.09	38.08	63.28	43.84	43.46	413.76

Table 2: Results by model and hop. Overall scores are weighted by questions per hop; #Tok denotes mean generated tokens. Bold and underline denote the best and second-best scores per column.

Model / Strategy	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok
GPT-5
Oracle	48.75	78.82	56.14	67.53	237.13
+ One-Shot	49.18	78.02	55.95	63.46	249.55
+ Negative Constraints	52.29	80.40	58.16	66.43	254.86
Claude-Sonnet-4.5
Oracle	47.53	76.66	54.16	63.57	376.09

Table 3: Prompting results. One-Shot and Negative Constraints: mean over questions. Best scores per model are bold; second-best are underlined.

Effort	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok
Low	48.50	78.66	55.22	72.98	274.21
High	43.68	75.09	51.49	71.65	261.91

Table 4: GPT-5 reasoning effort (Oracle); mean over questions. Best scores are bold.

Model / Setting	$P_{\text{Prec}}$	$P_{\text{Rec}}$	PAS	Faithfulness	#Tok
GPT-5
Base	39.58	68.63	46.26	34.96	486.99
Noisy	45.88	79.67	54.63	67.09	345.58
Oracle	48.75	78.82	56.14	67.53	237.13
Claude-Sonnet-4.5
Base	38.08	63.28	43.84	43.46	413.76

Table 5: Noisy results: mean over questions. Best scores per model are bold; second-best are underlined.

Model Name	Answer Relevancy Settings
Model Name	Base	Noisy	Oracle
Claude-Sonnet-4.5	0.7309	0.7297	0.7650
GPT-5	0.6568	0.6363	0.6498
Gemma3-27B-IT	0.6067	0.6443	0.6981
Qwen3-32B	0.7287	0.7294	0.7282

Table 6: Answer Relevancy. Best scores per model are bold; second-best are underlined.

Figure 4: Qualitative analysis of failures in selective exclusion. The History case (left) adds unrequested entities, while the Biology case (right) adds details beyond the requested sequence. Although factually accurate , both responses violate explicit constraints by merging multiple valid perspectives .

Setting	Claude- Sonnet-4.5	Qwen3-32B	Gemma3-27B
Base	38.08 / 63.28	39.91 / 68.02	38.53 / 65.13
RAG BM25	31.0 / 59.3	39.4 / 70.0	33.7 / 64.7
RAG dense	34.4 / 63.7	34.1 / 65.6	36.5 / 60.8
Noisy	38.74 / 69.96	42.86 / 76.92	44.03 / 73.17
Oracle	47.53 / 76.66	46.98 / 77.91	46.23 / 73.63

Table 7: Retrieval precision/recall with Noisy’s document count. Best scores per model are bold; second-best are underlined.

Model	Recall drop all-or-nothing / partial	Precision change all-or-nothing / partial
GPT-5	$-36.6$ / $-12.8$	$-0.7$ / $-6.0$
Qwen3-32B	$-31.7$ / $-12.1$	$+1.2$ / $+4.5$
Gemma3-27B	$-34.8$ / $-15.8$	$\pm 0.0$ / $+3.9$
Average	$-34.4$ / $-13.6$	$\approx$ flat

Table 8: Oracle recall drop and precision change, 1-hop to 3-hop (same Gemini judge).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of semantic perspective types across domains. Fine-grained relationships are aggregated into high-level categories. The distributions show the range of perspective types present in each domain.

Domain	Category	Representative Question
Physics	Sequential and Procedural	Describe the logical sequence connecting Newton’s second law of motion and the conservation of momentum, focusing on which concept must be established first to derive the other.
	Causal and Explanatory	Focusing on the underlying mechanism, describe how the vector summation of forces provides a framework for understanding an object’s state of constant velocity.
	Taxonomic and Hierarchical	Explain the connection between mechanical energy, conservation of momentum, Newton’s second law, and Newton’s third law, focusing on the hierarchical relationship.
Biology	Contrast and Alternatives	Focusing on their fundamental objectives and scope, explain what makes the process of gene transcription distinct from DNA replication.
	Input and Dependency	Considering chemiosmosis, the evolution of aerobic respiration, and fermentation, explain their relationship by focusing on the essential resources one process provides for another to function.
	Process Flow and Sequence	Considering Photosynthesis, Glycolysis, and the metabolism of glucose, explain their relationship by focusing on the procedural sequence where one action is part of a larger workflow.

Appendix

Table 9: Representative questions from the benchmark dataset, categorized by domain and semantic perspectives.

Figure 6: Prompt template for extraction of single-fact sentences and entities (Section 2.3.1 ). The LLM decomposes each text chunk $T$ into single-fact sentences $\mathcal{F}=\{f_{1},\ldots,f_{n}\}$ where each $f_{i}$ contains exactly one discrete piece of information, and pairs every $f_{i}$ with its key entities $e_{i}$ . The prompt requires pronoun resolution and 1:1 semantic preservation so that each $f_{i}$ remains self-contained when later (i) grouped into semantic clusters in Section 2.3.2 and (ii) served as a reference document during evaluation. The two outputs feed the next two construction stages in complementary ways: the single-fact sentences seed the density-based clustering of Section 2.3.2 , while the entities $e_{i}$ contribute the shared-entity signal $S_{\text{ent}}$ used by the edge-scoring heuristic in Section 2.3.3 .

Figure 7: Prompt template for representative summary generation (Section 2.3.2 ). After single-fact sentences in $\mathcal{F}$ are grouped into cohesive clusters $\{C_{1},\ldots,C_{k}\}$ , each cluster $C_{j}$ needs an explicit concept label to function as a graph node. The LLM synthesizes the facts in $C_{j}$ into a single representative summary $s_{j}$ that names the shared explanatory focus—for example, condensing facts about inflation and civil wars into “Internal Crisis” as in Section 2.3.2 . The instructions require synthesis rather than enumeration and forbid information not present in the input, so that $s_{j}$ faithfully labels $C_{j}$ . Each $(C_{j},s_{j})$ pair becomes a semantic node $v_{j}\in\mathcal{V}$ , on top of which edges and perspectives are constructed in Sections 2.3.3 and 2.3.4 .

Figure 8: Prompt template for edge generation (Section 2.3.3 ). For each top-scoring node pair $(v_{i},v_{j})$ that passes the heuristic relevance score $\text{Score}(v_{i},v_{j})=S_{\text{ent}}+S_{\text{prox}}+S_{\text{sem}}$ , the LLM receives the two representative summaries with supporting facts and emits one directed labeled edge per explanatory relationship it identifies. The prompt requires (i) bidirectional search ( $v_{i}\rightarrow v_{j}$ and $v_{j}\rightarrow v_{i}$ ), (ii) exhaustive enumeration rather than a single best label, and (iii) grounding each relation in the provided evidence, so a node pair may carry multiple edges with distinct labels in $\mathcal{E}$ . This is what lets a single endpoint pair $(v_{s},v_{e})$ later admit multiple valid reasoning paths $\Pi(v_{s}\rightarrow v_{e})$ —e.g., “External Invasion,” “Political Collapse,” and “Economic Collapse” from “Military Decay” to “Fall of Western Rome” in Figure 2 .

Figure 9: Prompt template for perspective-targeted question generation (Section 2.3.4 ). Given the valid paths $\Pi(v_{s}\rightarrow v_{e})$ between an endpoint pair, the LLM groups paths that share the same explanatory focus into a perspective $\mathcal{P}_{i}\subseteq\Pi(v_{s}\rightarrow v_{e})$ (e.g., causal, comparative, structural), generates one question $q_{i}$ per perspective, and records the subset $\Pi_{\mathcal{P}}^{\text{target}}$ of paths that validate it. The prompt forbids using the reason fields of paths and direct synonyms of relation labels. It requires general conceptual hints to indicate the target perspective. Full step-by-step formalism is in Appendix D .

Figure 10: Answer generation prompt for Base and Oracle settings (Appendix G ). The evaluated model receives a perspective-targeted question $q_{i}$ produced by the Step 4 prompt (Figure 9 ) and is instructed to return a coherent narrative explanation focused exclusively on the requested perspective, not an enumerated list of facts. The template branches on the [CONTEXT] slot: when empty, the model answers from parametric knowledge alone (Base setting); when populated with the ground-truth single-fact sentences along target paths $\Pi_{\mathcal{P}}^{\text{target}}$ , the model must ground its answer strictly in those reference documents (Oracle setting).

Figure 11: LLM-as-a-Judge prompt for the Perspective Alignment Score (Section 3.2 ). Following the CheckEval-style checklist formulation Lee et al. (2025) , the judge does not score a response holistically. It iterates through every reasoning path $\pi\in\Pi(q)$ in the supplied Valid_Paths_List and, for each path, returns an all-or-nothing judgment of whether the response semantically reproduces all of its constituent edges. The resulting identified set $A=\{\pi\in\Pi(q):\text{Judge}(\pi,\text{response})=\text{present}\}$ yields perspective precision $P_{\text{Prec}}=|A\cap T|/|A|$ , recall $P_{\text{Rec}}=|A\cap T|/|T|$ , and their harmonic mean PAS. Treating each path as an atomic unit is a deliberate granularity choice; sub-path violations are discussed in the Limitations.

Figure 12: LLM-as-a-Judge prompt for Faithfulness (Section 3.2 ). Following the standard claim-level verification protocol Es et al. (2024) , the prompt is organized into two stages. Stage 1 (Deconstruct): the judge breaks the response into a list of discrete factual claims ( claims_made ). Stage 2 (Verify): each claim is checked against the Ground_Truth_Sources —the ground-truth single-fact sentences drawn from target paths $\Pi_{\mathcal{P}}^{\text{target}}$ —and marked Yes (with the exact supporting single-fact sentence) or No (unsupported). Faithfulness is the proportion of supported claims. In the Oracle setting these same documents are exposed to the model during generation; in the Base setting they are used only post-hoc to verify the claims in the response (Section 3.2 ).

Benchmark	What it measures	Violation detected by	Nature of excluded content
IFEval Zhou et al. (2023)	Verifiable format and keyword constraints	Rules (code)	Keyword exclusion
FollowBench Jiang et al. (2024)	Multi-level constraint following	Rules + LLM	Mostly N/A
InfoBench Qin et al. (2024)	Decomposed atomic requirements	LLM judge	N/A
ComplexBench Wen et al. (2024)	Composed multi-constraint following	Rules + LLM	Partial (semantic)
EIFBENCH Zou et al. (2025)	Complex multi-task following with exclusion	LLM judge	Specific mentions
RealInstruct (DeCRIM) Ferraz et al. (2024)	Real user instructions with multiple constraints	Human + LLM	Explicit negative constraints

Appendix

Table 10: Comparison with existing benchmarks. N/A indicates that no separate exclusion target is specified in this comparison.

Comparison	Decision unit	Agreement	Cohen’s $\kappa$
Human / Gemini	220 path-level decisions	82.7%	0.64
GPT-5.6 Terra / Gemini	Path-level decisions	82.3%	0.645

Appendix

Table 11: Agreement at the path-level Yes/No unit.

Model	Precision	Recall	PAS
Base
Claude-Sonnet-4.5	38.08 / 36.0	63.28 / 63.9	43.84 / 42.8
Qwen3-32B	39.91 / 35.3	68.02 / 62.3	46.89 / 41.0
Gemma3-27B	38.53 / 36.7	65.13 / 65.6	44.48 / 43.0
Oracle
Claude-Sonnet-4.5	47.53 / 44.6	76.66 / 71.8	54.16 / 51.4

Appendix

Table 12: PAS evaluation under two judges. Each value pair reports Gemini / GPT-5.6 Terra. GPT-5.6 Terra re-scores the same responses.

Model	Gap (target GT)	Gap (full GT)	Change
GPT-5	32.6	32.1	$-0.5$
Claude-Sonnet-4.5	20.1	19.2	$-0.9$
Gemma3-27B	17.2	15.4	$-1.8$
Qwen3-32B	11.0	15.1	$+4.1$

Appendix

Table 13: Oracle

-

Base Faithfulness gap. Target GT uses target-path sentences; full GT includes all sentences in the question’s graph. Change is full GT

-

target GT.

Model	Base	Oracle
GPT-5	39.58 / 38.2	48.75 / 43.2
Claude-Sonnet-4.5	38.08 / 38.2	47.53 / 48.9
Qwen3-32B	39.91 / 41.2	46.98 / 43.7
Gemma3-27B	38.53 / 41.2	46.23 / 41.5

Appendix

Table 14: Precision on the original evaluation / unanimous items. Right values report scores on items where all three annotators agreed on distractor validity under the same scoring.

Metric	Cover-all (no exclusion)	Ten evaluated models
Precision	41.4	27–49
Recall	100	42–79
PAS	57.6	30–56

Appendix

Table 15: Cover-all baseline and model score ranges across Base and Oracle settings.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 396 papers

Apr 13, 2026

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not reveal the quality of the reasoning used to produce it. This highlights a fundamental limitation of outcome-based evaluation: models may arrive at correct answers through flawed reasoning, and models with substantially different reasoning capabilities can nevertheless exhibit similar benchmark accuracy, for example due to memorization or over-optimization. In this paper, we ask: given existing benchmarks, can we move beyond outcome-based evaluation to assess the quality of reasoning itself? We seek metrics that (1) differentiate models with similar accuracy and (2) are robust to variations in input prompts and generation configurations. To this end, we propose a reasoning score that evaluates reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality. A remaining question is how to aggregate this score across multiple sampled traces. Naively averaging them is undesirable, particularly in long-horizon settings, where the number of possible trajectories grows rapidly, and low-confidence correct traces are more likely to be coincidental. To address this, we introduce the Filtered Reasoning Score (FRS), which computes reasoning quality using only the top-K% most confident traces. Evaluating with FRS, models that are indistinguishable under standard accuracy exhibit significant differences in reasoning quality. Moreover, models with higher FRS on one benchmark tend to perform better on other reasoning benchmarks, in both accuracy and reasoning quality. Together, these findings suggest that FRS complements accuracy by capturing a model's transferable reasoning capabilities. We open source our evaluation codebase: https://github.com/HumainLab/filtered_reasoning_score_evaluation.

Manas Pathak, Xingyao Chen, Shuozhe Li +2

University of Texas at Austin

LLM Evaluation Language Model Generation Evaluation Reasoning Evaluation Reasoning Trace Analysis Language Model Reasoning Evaluation

Ask PDF

Aug 24, 2026

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."

Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4

Factual Knowledge in Language Models LLM Evaluation LLM Reliability Factual QA Adversarial Robustness of Language Models

Ask PDF

Aug 10, 2026

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturate".

Siyang Wu, Yibo Jiang, Bryon Aragam

Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA

Prompt Sensitivity LLM Evaluation Language Model Generation Evaluation Reasoning Consistency in Language Models LLM Reliability Cross-Lingual Consistency in Language Models

Ask PDF

Abstract

Language Model Generation Evaluation

Explore similar work

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility