Large Language Models often generate factually correct responses that fail to maintain user-requested perspectives, blending multiple perspectives and forcing users to filter through unwanted information---a perspective control failure that factual accuracy metrics cannot detect. We introduce PerspectiveBench, a benchmark quantifying perspective alignment by constructing scenarios where multiple valid reasoning paths coexist, and propose the Perspective Alignment Score (PAS) to measure adherence to specified perspectives. Experiments reveal that state-of-the-art models successfully include target content but fail to exclude non-requested perspectives. Providing documents through RAG improves factual grounding but does not resolve perspective drift, as models prioritize comprehensive coverage over strict user constraints. Code and data are available at https://github.com/sunghoon014/PerspectiveBench.
Figures & tables
Figure 1: Perspective misalignment problem.
Figure 2: The PerspectiveBench data construction pipeline: From atomic fact extraction and semantic clustering to graph-based reasoning path and perspective-targeted QA generation.
Domain
Hop
Avg. Total
Avg. Target
Avg. Distractor
Num. Qs
Physics
1
2.77
1.06
1.71
114
2
6.20
2.60
3.60
161
3
7.47
3.60
3.86
58
Biology
1
3.34
1.23
2.11
103
2
5.50
2.35
3.15
164
3
7.25
2.92
4.33
61
Table 1: Benchmark statistics by domain and hop.
Figure 3: Semantic similarity (bge-m3). Target and distractor paths overlap.
Model
Setting
1-hop
2-hop
3-hop
Overall
PPrec
PRec
PAS
Faithfulness
#Tok
PPrec
PRec
PAS
Faithfulness
#Tok
PPrec
PRec
PAS
Faithfulness
#Tok
PPrec
PRec
PAS
Faithfulness
#Tok
GPT-5
Base
44.37
92.50
56.13
33.06
464.92
37.33
58.66
42.13
35.56
489.89
35.90
46.79
37.27
37.26
524.71
39.58
68.63
46.26
34.96
486.99
Oracle
48.75
97.50
61.55
68.48
206.17
49.00
71.33
54.41
65.84
241.76
48.07
60.89
49.74
70.23
288.52
48.75
78.82
56.14
67.53
237.13
GPT-4.1
Base
51.50
96.67
63.80
51.88
301.58
39.33
44.00
38.29
48.16
331.95
31.41
39.74
32.75
51.92
361.98
42.29
61.94
46.39
50.12
326.32
Oracle
50.16
97.50
62.68
66.64
270.59
39.00
56.00
43.24
62.24
286.85
46.31
65.06
48.93
68.20
351.52
44.21
72.26
51.10
64.82
292.14
Claude-Sonnet-4.5
Base
43.87
90.83
56.12
44.67
405.14
35.00
49.33
37.06
42.10
413.23
34.62
44.87
37.19
44.71
433.09
38.08
63.28
43.84
43.46
413.76
Table 2: Results by model and hop. Overall scores are weighted by questions per hop; #Tok denotes mean generated tokens. Bold and underline denote the best and second-best scores per column.
Model / Strategy
PPrec
PRec
PAS
Faithfulness
#Tok
GPT-5
Oracle
48.75
78.82
56.14
67.53
237.13
+ One-Shot
49.18
78.02
55.95
63.46
249.55
+ Negative Constraints
52.29
80.40
58.16
66.43
254.86
Claude-Sonnet-4.5
Oracle
47.53
76.66
54.16
63.57
376.09
Table 3: Prompting results. One-Shot and Negative Constraints: mean over questions. Best scores per model are bold; second-best are underlined.
Effort
PPrec
PRec
PAS
Faithfulness
#Tok
Low
48.50
78.66
55.22
72.98
274.21
High
43.68
75.09
51.49
71.65
261.91
Table 4: GPT-5 reasoning effort (Oracle); mean over questions. Best scores are bold.
Model / Setting
PPrec
PRec
PAS
Faithfulness
#Tok
GPT-5
Base
39.58
68.63
46.26
34.96
486.99
Noisy
45.88
79.67
54.63
67.09
345.58
Oracle
48.75
78.82
56.14
67.53
237.13
Claude-Sonnet-4.5
Base
38.08
63.28
43.84
43.46
413.76
Table 5: Noisy results: mean over questions. Best scores per model are bold; second-best are underlined.
Model Name
Answer Relevancy Settings
Base
Noisy
Oracle
Claude-Sonnet-4.5
0.7309
0.7297
0.7650
GPT-5
0.6568
0.6363
0.6498
Gemma3-27B-IT
0.6067
0.6443
0.6981
Qwen3-32B
0.7287
0.7294
0.7282
Table 6: Answer Relevancy. Best scores per model are bold; second-best are underlined.
Figure 4: Qualitative analysis of failures in selective exclusion. The History case (left) adds unrequested entities, while the Biology case (right) adds details beyond the requested sequence. Although factually accurate , both responses violate explicit constraints by merging multiple valid perspectives .
Setting
Claude- Sonnet-4.5
Qwen3-32B
Gemma3-27B
Base
38.08 / 63.28
39.91 / 68.02
38.53 / 65.13
RAG BM25
31.0 / 59.3
39.4 / 70.0
33.7 / 64.7
RAG dense
34.4 / 63.7
34.1 / 65.6
36.5 / 60.8
Noisy
38.74 / 69.96
42.86 / 76.92
44.03 / 73.17
Oracle
47.53 / 76.66
46.98 / 77.91
46.23 / 73.63
Table 7: Retrieval precision/recall with Noisy’s document count. Best scores per model are bold; second-best are underlined.
Model
Recall drop all-or-nothing / partial
Precision change all-or-nothing / partial
GPT-5
−36.6 / −12.8
−0.7 / −6.0
Qwen3-32B
−31.7 / −12.1
+1.2 / +4.5
Gemma3-27B
−34.8 / −15.8
±0.0 / +3.9
Average
−34.4 / −13.6
≈ flat
Table 8: Oracle recall drop and precision change, 1-hop to 3-hop (same Gemini judge).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of semantic perspective types across domains. Fine-grained relationships are aggregated into high-level categories. The distributions show the range of perspective types present in each domain.
Domain
Category
Representative Question
Physics
Sequential and Procedural
Describe the logical sequence connecting Newton’s second law of motion and the conservation of momentum, focusing on which concept must be established first to derive the other.
Causal and Explanatory
Focusing on the underlying mechanism, describe how the vector summation of forces provides a framework for understanding an object’s state of constant velocity.
Taxonomic and Hierarchical
Explain the connection between mechanical energy, conservation of momentum, Newton’s second law, and Newton’s third law, focusing on the hierarchical relationship.
Biology
Contrast and Alternatives
Focusing on their fundamental objectives and scope, explain what makes the process of gene transcription distinct from DNA replication.
Input and Dependency
Considering chemiosmosis, the evolution of aerobic respiration, and fermentation, explain their relationship by focusing on the essential resources one process provides for another to function.
Process Flow and Sequence
Considering Photosynthesis, Glycolysis, and the metabolism of glucose, explain their relationship by focusing on the procedural sequence where one action is part of a larger workflow.
Appendix
Table 9: Representative questions from the benchmark dataset, categorized by domain and semantic perspectives.
Figure 6: Prompt template for extraction of single-fact sentences and entities (Section 2.3.1 ). The LLM decomposes each text chunk T into single-fact sentences F={f1,…,fn} where each fi contains exactly one discrete piece of information, and pairs every fi with its key entities ei . The prompt requires pronoun resolution and 1:1 semantic preservation so that each fi remains self-contained when later (i) grouped into semantic clusters in Section 2.3.2 and (ii) served as a reference document during evaluation. The two outputs feed the next two construction stages in complementary ways: the single-fact sentences seed the density-based clustering of Section 2.3.2 , while the entities ei contribute the shared-entity signal Sent used by the edge-scoring heuristic in Section 2.3.3 .
Figure 7: Prompt template for representative summary generation (Section 2.3.2 ). After single-fact sentences in F are grouped into cohesive clusters {C1,…,Ck} , each cluster Cj needs an explicit concept label to function as a graph node. The LLM synthesizes the facts in Cj into a single representative summary sj that names the shared explanatory focus—for example, condensing facts about inflation and civil wars into “Internal Crisis” as in Section 2.3.2 . The instructions require synthesis rather than enumeration and forbid information not present in the input, so that sj faithfully labels Cj . Each (Cj,sj) pair becomes a semantic node vj∈V , on top of which edges and perspectives are constructed in Sections 2.3.3 and 2.3.4 .
Figure 8: Prompt template for edge generation (Section 2.3.3 ). For each top-scoring node pair (vi,vj) that passes the heuristic relevance score Score(vi,vj)=Sent+Sprox+Ssem , the LLM receives the two representative summaries with supporting facts and emits one directed labeled edge per explanatory relationship it identifies. The prompt requires (i) bidirectional search ( vi→vj and vj→vi ), (ii) exhaustive enumeration rather than a single best label, and (iii) grounding each relation in the provided evidence, so a node pair may carry multiple edges with distinct labels in E . This is what lets a single endpoint pair (vs,ve) later admit multiple valid reasoning paths Π(vs→ve) —e.g., “External Invasion,” “Political Collapse,” and “Economic Collapse” from “Military Decay” to “Fall of Western Rome” in Figure 2 .
Figure 9: Prompt template for perspective-targeted question generation (Section 2.3.4 ). Given the valid paths Π(vs→ve) between an endpoint pair, the LLM groups paths that share the same explanatory focus into a perspective Pi⊆Π(vs→ve) (e.g., causal, comparative, structural), generates one question qi per perspective, and records the subset ΠPtarget of paths that validate it. The prompt forbids using the reason fields of paths and direct synonyms of relation labels. It requires general conceptual hints to indicate the target perspective. Full step-by-step formalism is in Appendix D .
Figure 10: Answer generation prompt for Base and Oracle settings (Appendix G ). The evaluated model receives a perspective-targeted question qi produced by the Step 4 prompt (Figure 9 ) and is instructed to return a coherent narrative explanation focused exclusively on the requested perspective, not an enumerated list of facts. The template branches on the [CONTEXT] slot: when empty, the model answers from parametric knowledge alone (Base setting); when populated with the ground-truth single-fact sentences along target paths ΠPtarget , the model must ground its answer strictly in those reference documents (Oracle setting).
Figure 11: LLM-as-a-Judge prompt for the Perspective Alignment Score (Section 3.2 ). Following the CheckEval-style checklist formulation Lee et al. (2025) , the judge does not score a response holistically. It iterates through every reasoning path π∈Π(q) in the supplied Valid_Paths_List and, for each path, returns an all-or-nothing judgment of whether the response semantically reproduces all of its constituent edges. The resulting identified set A={π∈Π(q):Judge(π,response)=present} yields perspective precision PPrec=∣A∩T∣/∣A∣ , recall PRec=∣A∩T∣/∣T∣ , and their harmonic mean PAS. Treating each path as an atomic unit is a deliberate granularity choice; sub-path violations are discussed in the Limitations.
Figure 12: LLM-as-a-Judge prompt for Faithfulness (Section 3.2 ). Following the standard claim-level verification protocol Es et al. (2024) , the prompt is organized into two stages. Stage 1 (Deconstruct): the judge breaks the response into a list of discrete factual claims ( claims_made ). Stage 2 (Verify): each claim is checked against the Ground_Truth_Sources —the ground-truth single-fact sentences drawn from target paths ΠPtarget —and marked Yes (with the exact supporting single-fact sentence) or No (unsupported). Faithfulness is the proportion of supported claims. In the Oracle setting these same documents are exposed to the model during generation; in the Base setting they are used only post-hoc to verify the claims in the response (Section 3.2 ).
Benchmark
What it measures
Violation detected by
Nature of excluded content
IFEval Zhou et al. (2023)
Verifiable format and keyword constraints
Rules (code)
Keyword exclusion
FollowBench Jiang et al. (2024)
Multi-level constraint following
Rules + LLM
Mostly N/A
InfoBench Qin et al. (2024)
Decomposed atomic requirements
LLM judge
N/A
ComplexBench Wen et al. (2024)
Composed multi-constraint following
Rules + LLM
Partial (semantic)
EIFBENCH Zou et al. (2025)
Complex multi-task following with exclusion
LLM judge
Specific mentions
RealInstruct (DeCRIM) Ferraz et al. (2024)
Real user instructions with multiple constraints
Human + LLM
Explicit negative constraints
Appendix
Table 10: Comparison with existing benchmarks. N/A indicates that no separate exclusion target is specified in this comparison.
Comparison
Decision unit
Agreement
Cohen’s κ
Human / Gemini
220 path-level decisions
82.7%
0.64
GPT-5.6 Terra / Gemini
Path-level decisions
82.3%
0.645
Appendix
Table 11: Agreement at the path-level Yes/No unit.
Model
Precision
Recall
PAS
Base
Claude-Sonnet-4.5
38.08 / 36.0
63.28 / 63.9
43.84 / 42.8
Qwen3-32B
39.91 / 35.3
68.02 / 62.3
46.89 / 41.0
Gemma3-27B
38.53 / 36.7
65.13 / 65.6
44.48 / 43.0
Oracle
Claude-Sonnet-4.5
47.53 / 44.6
76.66 / 71.8
54.16 / 51.4
Appendix
Table 12: PAS evaluation under two judges. Each value pair reports Gemini / GPT-5.6 Terra. GPT-5.6 Terra re-scores the same responses.
Model
Gap (target GT)
Gap (full GT)
Change
GPT-5
32.6
32.1
−0.5
Claude-Sonnet-4.5
20.1
19.2
−0.9
Gemma3-27B
17.2
15.4
−1.8
Qwen3-32B
11.0
15.1
+4.1
Appendix
Table 13: Oracle − Base Faithfulness gap. Target GT uses target-path sentences; full GT includes all sentences in the question’s graph. Change is full GT − target GT.
Model
Base
Oracle
GPT-5
39.58 / 38.2
48.75 / 43.2
Claude-Sonnet-4.5
38.08 / 38.2
47.53 / 48.9
Qwen3-32B
39.91 / 41.2
46.98 / 43.7
Gemma3-27B
38.53 / 41.2
46.23 / 41.5
Appendix
Table 14: Precision on the original evaluation / unanimous items. Right values report scores on items where all three annotators agreed on distractor validity under the same scoring.
Metric
Cover-all (no exclusion)
Ten evaluated models
Precision
41.4
27–49
Recall
100
42–79
PAS
57.6
30–56
Appendix
Table 15: Cover-all baseline and model score ranges across Base and Oracle settings.
Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not reveal the quality of the reasoning used to produce it. This highlights a fundamental limitation of outcome-based evaluation: models may arrive at correct answers through flawed reasoning, and models with substantially different reasoning capabilities can nevertheless exhibit similar benchmark accuracy, for example due to memorization or over-optimization. In this paper, we ask: given existing benchmarks, can we move beyond outcome-based evaluation to assess the quality of reasoning itself? We seek metrics that (1) differentiate models with similar accuracy and (2) are robust to variations in input prompts and generation configurations. To this end, we propose a reasoning score that evaluates reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality. A remaining question is how to aggregate this score across multiple sampled traces. Naively averaging them is undesirable, particularly in long-horizon settings, where the number of possible trajectories grows rapidly, and low-confidence correct traces are more likely to be coincidental. To address this, we introduce the Filtered Reasoning Score (FRS), which computes reasoning quality using only the top-K% most confident traces. Evaluating with FRS, models that are indistinguishable under standard accuracy exhibit significant differences in reasoning quality. Moreover, models with higher FRS on one benchmark tend to perform better on other reasoning benchmarks, in both accuracy and reasoning quality. Together, these findings suggest that FRS complements accuracy by capturing a model's transferable reasoning capabilities. We open source our evaluation codebase: https://github.com/HumainLab/filtered_reasoning_score_evaluation.
Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."
Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4
Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturate".
Siyang Wu, Yibo Jiang, Bryon Aragam
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA