Contracts, technical manuals, and online communities often give familiar words special meanings and define them in one place. A language model that reads a passage without that definition may use the usual meaning and give an answer that sounds right but is wrong. We study this with multiple-choice questions whose correct answers depend on a local definition. The questions come from a synthetic vocabulary, commercial contracts, Reddit community glossaries, and the documentation of a decentralized exchange. We ask three language models each question with and without the definition. Without using the answer key, we then test whether their repeated answers and stated confidence reveal their own errors. Separately, we give a small open model the answer key and test whether it can predict which questions larger models get wrong. We report three findings. (1) Familiar words that a source redefines cause the most errors. Models often repeat the same wrong answer, and neither repeated answers nor stated confidence reliably flag it. (2) The source definition fixes most errors on synthetic questions, but a definition drawn from the model's own prior knowledge does not. In real contracts, the source definition fixes only about half of the errors. (3) Given an answer key, a small model can rank questions by how likely larger models are to get them wrong. These results call for tests that check answers against the full source, including its local definitions.
Figures & tables
Error type
Answer judged against
Needed evidence in the input
Factuality error
World knowledge
Not required
Faithfulness error
Supplied input
Present
Knowledge conflict
Input versus learned knowledge
Present, contradicts the prior
Missing local definition
Full source document only
Absent
Table 1: Error types by the standard an answer is judged against. The first three categories are used in hallucination and knowledge-conflict research ( Wang et al., 2025c ; Longpre et al., 2021 ; Xu et al., 2024 ) .
Situation
Missing or misleading information
Possible remedy
Meaning conflict
The usual meaning supports a different answer from the local definition.
Supply the source definition and its scope.
Scope unspecified
The general concept fits, but its members or boundary in this document are unknown.
Supply the local boundary or binding.
Other information
The needed fact or rule is outside the definition.
Retrieve the relevant evidence.
Table 2: Three possible reasons an answer may need local information. These are interpretive cases, not annotated question categories.
Term
Local definition
Question (abridged)
Answer implied by the definition
Risk
Products
dietary supplements manufactured within the United States
Which products must the endorser have used to give honest findings?
only U.S.-made supplements
97th
Trademarks
the trademarks and trade names identified in Exhibit A
Must the distributor report a confusable third-party name not in Exhibit A?
report only names listed in Exhibit A
99.7th
Term
the period from the Signing Date to the End Date
During which period must the required insurance stay in force?
from signing to the stated end date, not until termination
98th
Table 3: Examples from the contract validation set. GPT-5.6-terra answers each question incorrectly without the definition and correctly with it. The usual meanings suggest broader readings than the contracts allow. Risk percentiles refer to the 6,529 scored questions.
Figure 1: (a) Error rates without definitions for the three synthetic term classes. Errors are most common on redefined common words. (b) Share of GPT-6-astra’s errors that another reader makes on the same questions. Under independence, the expected share is that reader’s overall error rate on those questions. For DeFi, the comparison counts an error by either DeepSeek V4.1 Flash or GPT-5.6-terra.
Figure 2: (a) Error-ranking AUROC for DeepSeek V4 Flash on 456 synthetic questions. We obtain 95% intervals by resampling terms. Gray bars show signals from ten sampled answers and stated confidence, which use no key. The light bar shows key-free uncertainty for the 7B proxy. The dark bar shows the proxy score R (Section 7 ), which uses the answer key. The dotted line marks chance. (b) How well Jev’s confidence ranks its own errors, with no definition and with a drafted gloss. (c) DeepSeek V4 Flash accuracy with no definition, the source definition, a drafted gloss, or a prior gloss. On the right, the lexical screen finds no option singled out by word overlap with any of the three notes.
Corpus
Reader
n
Error, no def.
Error, def.
R AUROC [95% CI]
Freq.
Synthetic
DeepSeek V4 Flash
456
0.285
0.029
0.848 [0.811, 0.883]
0.540
Synthetic
GPT-6-astra
456
0.182
0.018
0.809 [0.763, 0.856]
0.629
CUAD
GPT-5.6-terra
1,500
0.213
0.160
0.815 [0.789, 0.840]
0.507
Reddit
DeepSeek V4.1 Flash
1,391
0.301
0.185
0.808 [0.785, 0.835]
0.544
Reddit
GPT-6-astra
1,391
0.146
0.119
0.720 [0.676, 0.761]
0.588
DeFi
DeepSeek V4.1 Flash
379
0.282
0.193
0.809 [0.763, 0.856]
0.510
Table 4: Reader error without and with the definition text, and AUROC for ranking errors without the definition. R is the proxy score. Freq. is the better of term frequency and term rarity as a ranking score. The definition text is a drafted gloss, written from the source, for synthetic, Reddit, and DeFi, and extracted contract wording for CUAD. Rows differ in reader and question construction (Appendix A ), and CUAD is risk-stratified.
Figure 3: CUAD validation with GPT-5.6-terra as the reader. We sample 300 questions from each score bin. (a) Observed error rises with proxy score. The dotted line shows where the two values would be equal. The score ranks errors and does not estimate their probability. (b) Outcomes after adding the extracted contract definition. More errors are fixed than correct answers are made wrong, and about half of the initial errors remain.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Both key
Both other
Split answer
One supported
Neither supported
Synthetic
18
0
0
1
1
CUAD
13
1
1
4
1
Reddit
3
0
0
3
14
DeFi
5
0
0
7
8
Appendix
Table 5: Model source screening on 20 questions per corpus. “Both key” means both models gave a unique source-supported answer matching the stored key. “Both other” means they agreed on another answer. “Split answer” means two supported but different answers. “One supported” means one model judged the source sufficient and the other did not. “Neither supported” includes insufficient or ambiguous evidence. These are screening categories, not expert answer-key error rates.
Corpus
Key supported
Invalid or source-insufficient
Borderline
Reviewed
Synthetic
3
1
0
4
CUAD
2
6
1
9
Reddit
3
1
1
5
DeFi
2
6
1
9
Total
10
14
3
27
Appendix
Table 6: Source review of 27 selected diagnostic questions: 17 model-screen disagreements and ten stratified controls. Counts describe this enriched sample, not corpus-wide defect rates.
Setting
Items retained
R error AUROC
Conditional repair Δp AUROC
Synthetic / DeepSeek V4 Flash
455/456
.848 → .847
.983 → .983
Synthetic / GPT-6-astra
455/456
.809 → .809
.962 → .962
CUAD / GPT-5.6-terra
1,494/1,500
.815 → .817
.712 → .714
Reddit / DeepSeek V4.1 Flash
1,390/1,391
.808 → .808
.858 → .860
Reddit / GPT-6-astra
1,390/1,391
.720 → .720
.812 → .812
DeFi / DeepSeek V4.1 Flash
373/379
.809 → .808
.863 → .863
Appendix
Table 7: Deletion sensitivity after removing the 14 reviewed questions classified as clearly invalid or source-insufficient. Other questions retain their historical keys. This is a local sensitivity analysis, not a validated-only re-estimate. Δp AUROC targets repair among observed bare errors on the same question.
Corpus
Reader
Definition given to reader
Question drafting input
Synthetic
DeepSeek V4 Flash
Drafted gloss ( glossC )
Invented-term usage contexts. No source definition in drafting prompt
Synthetic
GPT-6-astra
Drafted gloss ( glossC )
Same synthetic questions
CUAD
GPT-5.6-terra
Source definition ( def_raw )
One definition and one contract usage per question
Reddit
DeepSeek V4.1 Flash
Drafted gloss ( glossC )
Community glossary entry and comment contexts
Reddit
GPT-6-astra
Drafted gloss ( glossC )
Same Reddit questions
DeFi
DeepSeek V4.1 Flash
Drafted gloss ( glossC )
Source glossary entry and documentation contexts
Appendix
Table 8: Reader settings in Table 4 . Synthetic DeepSeek calls used V4 Flash. Later calls used V4.1 Flash. DeepSeek documents the API IDs deepseek-v4-flash and deepseek-flash , respectively. “Source entry” describes drafting input and does not imply that the reader’s glossC arm used that entry.
Class
Questions
DeepSeek V4 Flash
GPT-6-astra
Redefined common words
156
69, 0.803 [0.739, 0.864]
57, 0.777 [0.697, 0.854]
Invented words
158
40, 0.779 [0.704, 0.849]
20, 0.720 [0.629, 0.819]
Usual-meaning words
142
21, 0.921 [0.870, 0.966]
6, 0.846 [0.710, 0.993]
Appendix
Table 9: Error-ranking R by designed synthetic term class. Reader cells give errors, AUROC [95% CI]. Both readers answer the same 456 questions. Intervals resample term records.
Corpus
Target reader
n
Proxy R
Other readers
Synthetic
DeepSeek V4 Flash
456
.848
.722
Synthetic
GPT-6-astra
456
.809
.804
Reddit
DeepSeek V4.1 Flash
1,391
.808
.640
Reddit
GPT-6-astra
1,391
.720
.736
DeFi
DeepSeek V4.1 Flash
379
.809
.750
DeFi
GPT-5.6-terra
379
.792
.867
Appendix
Table 10: AUROC for predicting a target reader’s bare errors on the same questions. “Other readers” is one observed reader error or the mean of two other reader errors. All models share the historical question and answer key.
Stability (761 questions)
Detection AUROC (456 questions)
Class
Errors
Same wrong
Majority wrong
Errors
Disagreement
Stated conf.
Proxy, no key
Proxy R
Redefined common words
168/461
74 (44%)
110 (65%)
69
0.557 [0.492, 0.622]
0.463 [0.376, 0.554]
0.537 [0.439, 0.636]
0.803 [0.738, 0.864]
Invented words
40/158
12 (30%)
23 (58%)
40
0.641 [0.570, 0.720]
0.715 [0.637, 0.785]
0.718 [0.615, 0.816]
0.779 [0.704, 0.848]
Usual-meaning words
21/142
7 (33%)
9 (43%)
21
0.603 [0.517, 0.692]
0.644 [0.552, 0.730]
0.810 [0.710, 0.897]
0.921 [0.871, 0.965]
All
229/761
93 (41%)
142 (62%)
130
0.616 [0.579, 0.656]
0.617 [0.558, 0.681]
0.689 [0.632, 0.750]
0.848 [0.812, 0.881]
Appendix
Table 11: Synthetic stability and error-detection AUROC by term class for DeepSeek V4 Flash. “Same wrong” counts greedy errors repeated identically in all ten samples. “Majority wrong” counts errors that survive majority voting.
DeepSeek Flash (V4/4.1)
GPT-6-astra
Corpus
Term class
n
Error
R AUROC
Error
R AUROC
Synthetic
Redefined common words
156
0.442
0.803
0.365
0.777
Synthetic
Invented words
158
0.253
0.779
0.127
0.720
Synthetic
Usual-meaning words
142
0.148
0.921
0.042
0.846
Reddit
Glossary term
851
0.343
0.781
0.148
0.659
Reddit
Control word
540
0.235
0.842
0.143
0.815
Appendix
Table 12: Bare error and proxy R AUROC by term class. Synthetic classes are fixed by the corpus design. Reddit glossary terms have a community entry, and control words are ordinary words from the same communities without one.
Setting
n
Repair AUROC R
Repair AUROC Δp
Error AUROC difference Δp−R , 95% CI
Synthetic / DeepSeek V4 Flash
456
0.833
0.810
[-0.168, -0.046]
Synthetic / GPT-6-astra
456
0.797
0.716
[-0.259, -0.066]
CUAD library / GPT-5.6-terra
1500
0.760
0.595
[-0.395, -0.302]
Reddit / DeepSeek V4.1 Flash
1391
0.744
0.709
[-0.323, -0.244]
Reddit / GPT-6-astra
1391
0.725
0.621
[-0.364, -0.213]
DeFi / DeepSeek V4.1 Flash
379
0.769
0.686
[-0.445, -0.251]
Appendix
Table 13: Prediction of observed repair. The last column concerns the separate bare-error target and gives a paired ranking difference. R and Δp are algebraically coupled (Appendix B ).
Setting
Δp error AUROC
Mean pd
Mean Δp
Within- R -decile AUROC
Synthetic / DeepSeek V4 Flash
0.745
.804 / .924
.496 / .246
.485
Synthetic / GPT-6-astra
0.656
.751 / .920
.450 / .287
.431
CUAD / GPT-5.6-terra
0.464
.407 / .714
.139 / .141
.398
Reddit / DeepSeek V4.1 Flash
0.520
.440 / .675
.163 / .131
.409
Reddit / GPT-6-astra
0.429
.405 / .639
.105 / .147
.359
DeFi / DeepSeek V4.1 Flash
0.458
.410 / .670
.116 / .130
.411
Appendix
Table 14: Why Δp can rank bare-reader errors below chance. Means are listed as bare-error / bare-correct items. The final column compares only error-correct pairs inside the same proxy- R decile, a descriptive approximate control for the common p0 term. It is not a causal decomposition.
Term score
Uses questions?
Terms (shifted / consistent)
Class AUROC
No-context definition distance
No
66 (38 / 28)
0.946
Domain-prior definition distance
No
157 (118 / 39)
0.977
Model self-report
No
79 (40 / 39)
0.984
Drafted-local/prior answer contrast
Yes
78 (40 / 38)
0.917
Mean Δp
Yes
78 (40 / 38)
0.783
Mean R
Yes
78 (40 / 38)
0.817
Appendix
Table 15: Distinguishing designed redefinitions from usual-meaning terms in the synthetic vocabulary, after excluding one source-definition-conflicted term. Each score uses all available terms for that score. Coverage differs and point estimates should not be ranked as if measured on one common sample. The class label was fixed by construction.
Corpus
No-context distance [95% interval]
Domain-prior distance
Self-report
Synthetic
0.452[0.260,0.602] (65)
0.300 (186)
0.465 (118)
Reddit
0.181[0.046,0.303] (211)
0.095 (217)
0.166 (217)
DeFi
−0.087[−0.361,0.186] (57)
0.101 (64)
−0.237 (64)
Appendix
Table 16: Spearman association of question-independent term scores with source-definition repair rate. Parentheses give available terms. Distance confidence intervals resample term records. There is no aligned glossG reader arm for this 204-term CUAD package. Only 54 terms have an uncontextualized prior gloss.
Setting
Bare failures
R
Δp
Other-item Δp ( n )
Synthetic / DeepSeek V4 Flash
130
0.445
0.983
0.914 (130)
Synthetic / GPT-6-astra
83
0.421
0.962
0.976 (83)
CUAD / GPT-5.6-terra
320
0.462
0.712
0.581 (190)
Reddit / DeepSeek V4.1 Flash
419
0.453
0.858
0.824 (418)
Reddit / GPT-6-astra
203
0.554
0.812
0.864 (203)
DeFi / DeepSeek V4.1 Flash
107
0.501
0.863
0.850 (107)
Appendix
Table 17: Predicting which of a reader’s errors the definition text repairs, as AUROC for ranking repaired above unrepaired errors among errors made without the definition. R and Δp use the same diagnostic question in the middle columns. The last column uses other scored questions of the same term record, with the available failed-question count in parentheses. CUAD uses the full proxy-scored frame for sibling estimates. Other rows use their available scored siblings. The CUAD 95% interval is [0.494,0.661] .
Corpus
n
Bare
Source glossG
Drafted glossC
Prior glossM
Synthetic
761
.699
.966 (209/6)
.949 (196/6)
.708 (86/79)
Reddit glossary
851
.657
.932 (243/9)
.790 (171/58)
.616 (82/117)
DeFi source-entry
250
.672
.940 (70/3)
.760 (42/20)
.740 (40/23)
Appendix
Table 18: DeepSeek Flash readers on matched questions under four conditions: V4 for synthetic and V4.1 for Reddit and DeFi. Cell values are accuracy, followed by repairs/harms relative to bare. glossG is the source definition, glossC the drafted gloss, and glossM the prior gloss, elicited under a domain prompt without the source examples. The synthetic row excludes two source-conflicted questions. Reddit and DeFi drafters saw the source definition.
Corpus
n
Cue
Shuffle
No-cue n
No-cue errors
No-cue AUROC: R / repair- Δp
Synthetic
456
.686
.067
143
.308 → .091
.879 / .950
Reddit
1,391
.185
.032
1,133
.296 → .210
.810 / .844
DeFi
379
.274
.086
275
.284 → .236
.803 / .791
Appendix
Table 19: Lexical-cue screen using the exact glossC text supplied to the archived DeepSeek Flash readers (V4 for synthetic, V4.1 for Reddit and DeFi). Synthetic values use the reader-aligned raw_outputs/v1_weights.jsonl , rather than a different same-named input copy. Error rates are bare → drafted-gloss.
Workload
n
Retrieval
Selected notes
All matched
Full glossary
CUAD
1,200
0.821
0.871
0.882
0.876
Reddit / all
1,391
0.866
0.925
n/a
0.907
Reddit / glossary
851
0.839
0.941
n/a
0.907
Reddit / control
540
0.907
0.898
n/a
0.907
Appendix
Table 20: In-sample hybrid-retrieval results on constructed questions. The risk aggregate includes the evaluated diagnostic item. The all-matched arm is unavailable for Reddit. The CUAD full glossary is capped at 30 entries. These arms are not matched-budget random or frequency policies.
Corpus / tier / subset
n
Gain, pp [95% CI]
Repair/harm
Selected/full units
CUAD / BM25 / all
1200
4.17 [2.24, 6.12]
94 / 44
135.8 / 529.2
CUAD / dense / all
1200
5.33 [3.49, 7.35]
102 / 38
79.1 / 529.2
CUAD / hybrid / all
1200
5.00 [3.34, 6.73]
93 / 33
126.6 / 529.2
Reddit / BM25 / all
1391
6.69 [4.45, 9.04]
112 / 19
39.4 / 2765.9
Reddit / BM25 / glossary
851
10.34 [7.01, 13.83]
97 / 9
46.5 / 3285.2
Reddit / BM25 / control
540
0.93 [-0.89, 2.78]
15 / 10
28.3 / 1947.5
Appendix
Table 21: Full RAG accounting: selected-definition policy relative to retrieval only. Repairs and harms are paired counts. Added definition units are inferred by subtracting identical retrieval contexts. They include glossary formatting.
Task
n
Arm
Score
Difference [95% CI]
CUAD F1
1661
rag_full
0.5270
+0.0029 [-0.0037, +0.0100]
CUAD F1
1661
rag_gated
0.5249
+0.0008 [-0.0051, +0.0073]
ContractNLI acc.
3060
note_gated
0.8173
+0.0059 [-0.0020, +0.0147]
ContractNLI acc.
3060
note_all
0.8222
+0.0108 [+0.0026, +0.0193]
Reddit / GPT
178
note_gated
0.5169
+0.0169 [-0.0382, +0.0566]
Reddit / GPT
178
note_all
0.5281
+0.0281 [-0.0715, +0.0909]
Appendix
Table 22: Public-task interventions. Metric changes are in raw F1/accuracy units, with paired cluster-bootstrap intervals. Reddit labels refer to post-term items.
Definition text
A direct: bare failures / repairs, AUROC [95% CI]
A to B: bare failures / repairs, AUROC [95% CI]
Drafted gloss
144 / 85, 0.833 [0.773, 0.882]
147 / 85, 0.811 [0.732, 0.893]
Community source entry
144 / 116, 0.665 [0.539, 0.788]
147 / 126, 0.600 [0.475, 0.717]
Appendix
Table 23: Reddit conditional repair with the version of the definition held fixed between proxy scoring and reader intervention. These historical questions were model drafted and are not independent future requests.
Model
Hugging Face commit (first 12 characters)
Qwen2.5-0.5B
060db6499f32
Qwen2.5-1.5B
8faed761d45a
Qwen2.5-3B
3aab1f1954e9
Qwen2.5-7B
d14972939875
Qwen2.5-14B
97e1e76335b7
Qwen2.5-14B-Instruct
cf98f3b3bbb4
Appendix
Table 24: Proxy model checkpoints. Each entry is the first 12 characters of the Hugging Face commit hash that identifies the exact weights we used. The full 40-character hashes are in the accompanying model records.
Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We ask what it takes to talk a model out of an answer it gets right on its own, and separate three factors that earlier studies vary together: where the advice sits in the prompt, who is said to have written it, and what the model has already written. On Qwen3.5-27B and four-option MMLU-Pro items, with the rest of the advice held fixed in each comparison, we find three things. First, position matters far more than the API channel, in opposite directions for a bare claim and an argument: a bare "the answer is X" does most harm just before the model answers, while the same claim with a false rationale does most harm before the question, where it raises error on items the model otherwise almost never misses from 1% to 67%; moving the claim between the system message and the user message at the same position changes error by a few points. Second, attributing the note to an authority rather than to another participant, and phrasing it assertively, each raise error by 10-27 points, while the two authorities we tested differ by under two. Third, replaying a correct worked answer before the advice protects it whether the text is shown as the model's own or as someone else's, and the same replay keeps wrong answers from being corrected by a bare correct note. The position result replicates under an independently written prompt. Code, items and all result tables are released at https://github.com/coolkillercat/safety.
Whether a language model's answer is correct often depends on context: the same value can be right under one jurisdiction, date, or legal definition and wrong under another. We introduce GT-FENG (General Theory of Factual Evaluation and Normative Grounding), the first general framework to unify contextual/temporal correctness and rule-based error attribution across law, policy, and other rule-governed domains. GT-FENG checks answers against the applicable rules and identifies documented rules that match incorrect answers. It shows that when multiple rules give the same value, the answer alone cannot tell which rule the model used. In Medicaid income-limit answers from two models, documenting the asked state's own records reduces the number of wrong answers that match only other states' values by 68-78%, and more than half of stable wrong answers then match two or more documented sources. GT-FENG provides a foundation for evaluating language models in rule-governed domains and for future benchmarks and evaluation toolkits. We release GT-FENG as an open-source Python package, gtfeng.
Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferring to the user. Yet the same models are far more compliant when a wrong answer is attributed to a verified source, which is how retrieval results, tool outputs, and grounded-search content often present information. We measure this gap across five open-weight families and three closed APIs. A single verified-source note endorsing a wrong answer flips 45-88% of baseline-correct responses in seven of eight models, and compliance rises with how authoritative the note sounds. Source deference and user agreement are not behaviorally interchangeable inside the model: on matched items with the same wrong answer, causal interventions can selectively suppress one without equally affecting the other. In three open-weight families, removing a fitted source direction lowers source compliance by 65-80 percentage points while removing a user or assistant direction has far smaller effects, and removing the user direction shows the reverse preference. A separately fitted intervention derived from source-versus-user cue activations moves compliance in both directions while leaving the prompt text unchanged. An authority direction fitted on trivia also transfers to PIQA and multi-turn SYCON dialogues without refitting, and removing it lowers wrong-source compliance by tens of percentage points in four of five families with no detected change in MMLU-Pro or GSM8K accuracy at our evaluation sizes. Source deference and user agreement therefore need separate evaluation.