When Local Definitions Are Missing: Stable Errors in Language Models
Organizations: University of Osaka · Nagoya University
Abstract
Contracts, technical manuals, and online communities often give familiar words special meanings and define them in one place. A language model that reads a passage without that definition may use the usual meaning and give an answer that sounds right but is wrong. We study this with multiple-choice questions whose correct answers depend on a local definition. The questions come from a synthetic vocabulary, commercial contracts, Reddit community glossaries, and the documentation of a decentralized exchange. We ask three language models each question with and without the definition. Without using the answer key, we then test whether their repeated answers and stated confidence reveal their own errors. Separately, we give a small open model the answer key and test whether it can predict which questions larger models get wrong. We report three findings. (1) Familiar words that a source redefines cause the most errors. Models often repeat the same wrong answer, and neither repeated answers nor stated confidence reliably flag it. (2) The source definition fixes most errors on synthetic questions, but a definition drawn from the model's own prior knowledge does not. In real contracts, the source definition fixes only about half of the errors. (3) Given an answer key, a small model can rank questions by how likely larger models are to get them wrong. These results call for tests that check answers against the full source, including its local definitions.
Figures & tables
| Error type | Answer judged against | Needed evidence in the input |
|---|---|---|
| Factuality error | World knowledge | Not required |
| Faithfulness error | Supplied input | Present |
| Knowledge conflict | Input versus learned knowledge | Present, contradicts the prior |
| Missing local definition | Full source document only | Absent |
| Situation | Missing or misleading information | Possible remedy |
|---|---|---|
| Meaning conflict | The usual meaning supports a different answer from the local definition. | Supply the source definition and its scope. |
| Scope unspecified | The general concept fits, but its members or boundary in this document are unknown. | Supply the local boundary or binding. |
| Other information | The needed fact or rule is outside the definition. | Retrieve the relevant evidence. |
| Term | Local definition | Question (abridged) | Answer implied by the definition | Risk |
|---|---|---|---|---|
| Products | dietary supplements manufactured within the United States | Which products must the endorser have used to give honest findings? | only U.S.-made supplements | 97th |
| Trademarks | the trademarks and trade names identified in Exhibit A | Must the distributor report a confusable third-party name not in Exhibit A? | report only names listed in Exhibit A | 99.7th |
| Term | the period from the Signing Date to the End Date | During which period must the required insurance stay in force? | from signing to the stated end date, not until termination | 98th |
| Corpus | Reader | Error, no def. | Error, def. | AUROC [95% CI] | Freq. | |
|---|---|---|---|---|---|---|
| Synthetic | DeepSeek V4 Flash | 456 | 0.285 | 0.029 | 0.848 [0.811, 0.883] | 0.540 |
| Synthetic | GPT-6-astra | 456 | 0.182 | 0.018 | 0.809 [0.763, 0.856] | 0.629 |
| CUAD | GPT-5.6-terra | 1,500 | 0.213 | 0.160 | 0.815 [0.789, 0.840] | 0.507 |
| DeepSeek V4.1 Flash | 1,391 | 0.301 | 0.185 | 0.808 [0.785, 0.835] | 0.544 | |
| GPT-6-astra | 1,391 | 0.146 | 0.119 | 0.720 [0.676, 0.761] | 0.588 | |
| DeFi | DeepSeek V4.1 Flash | 379 | 0.282 | 0.193 | 0.809 [0.763, 0.856] | 0.510 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Both key | Both other | Split answer | One supported | Neither supported |
|---|---|---|---|---|---|
| Synthetic | 18 | 0 | 0 | 1 | 1 |
| CUAD | 13 | 1 | 1 | 4 | 1 |
| 3 | 0 | 0 | 3 | 14 | |
| DeFi | 5 | 0 | 0 | 7 | 8 |
| Corpus | Key supported | Invalid or source-insufficient | Borderline | Reviewed |
|---|---|---|---|---|
| Synthetic | 3 | 1 | 0 | 4 |
| CUAD | 2 | 6 | 1 | 9 |
| 3 | 1 | 1 | 5 | |
| DeFi | 2 | 6 | 1 | 9 |
| Total | 10 | 14 | 3 | 27 |
| Setting | Items retained | error AUROC | Conditional repair AUROC |
|---|---|---|---|
| Synthetic / DeepSeek V4 Flash | 455/456 | .848 .847 | .983 .983 |
| Synthetic / GPT-6-astra | 455/456 | .809 .809 | .962 .962 |
| CUAD / GPT-5.6-terra | 1,494/1,500 | .815 .817 | .712 .714 |
| Reddit / DeepSeek V4.1 Flash | 1,390/1,391 | .808 .808 | .858 .860 |
| Reddit / GPT-6-astra | 1,390/1,391 | .720 .720 | .812 .812 |
| DeFi / DeepSeek V4.1 Flash | 373/379 | .809 .808 | .863 .863 |
| Corpus | Reader | Definition given to reader | Question drafting input |
|---|---|---|---|
| Synthetic | DeepSeek V4 Flash | Drafted gloss ( glossC ) | Invented-term usage contexts. No source definition in drafting prompt |
| Synthetic | GPT-6-astra | Drafted gloss ( glossC ) | Same synthetic questions |
| CUAD | GPT-5.6-terra | Source definition ( def_raw ) | One definition and one contract usage per question |
| DeepSeek V4.1 Flash | Drafted gloss ( glossC ) | Community glossary entry and comment contexts | |
| GPT-6-astra | Drafted gloss ( glossC ) | Same Reddit questions | |
| DeFi | DeepSeek V4.1 Flash | Drafted gloss ( glossC ) | Source glossary entry and documentation contexts |
| Class | Questions | DeepSeek V4 Flash | GPT-6-astra |
|---|---|---|---|
| Redefined common words | 156 | 69, 0.803 [0.739, 0.864] | 57, 0.777 [0.697, 0.854] |
| Invented words | 158 | 40, 0.779 [0.704, 0.849] | 20, 0.720 [0.629, 0.819] |
| Usual-meaning words | 142 | 21, 0.921 [0.870, 0.966] | 6, 0.846 [0.710, 0.993] |
| Corpus | Target reader | Proxy | Other readers | |
|---|---|---|---|---|
| Synthetic | DeepSeek V4 Flash | 456 | .848 | .722 |
| Synthetic | GPT-6-astra | 456 | .809 | .804 |
| DeepSeek V4.1 Flash | 1,391 | .808 | .640 | |
| GPT-6-astra | 1,391 | .720 | .736 | |
| DeFi | DeepSeek V4.1 Flash | 379 | .809 | .750 |
| DeFi | GPT-5.6-terra | 379 | .792 | .867 |
| Stability (761 questions) | Detection AUROC (456 questions) | |||||||
|---|---|---|---|---|---|---|---|---|
| Class | Errors | Same wrong | Majority wrong | Errors | Disagreement | Stated conf. | Proxy, no key | Proxy |
| Redefined common words | 168/461 | 74 (44%) | 110 (65%) | 69 | 0.557 [0.492, 0.622] | 0.463 [0.376, 0.554] | 0.537 [0.439, 0.636] | 0.803 [0.738, 0.864] |
| Invented words | 40/158 | 12 (30%) | 23 (58%) | 40 | 0.641 [0.570, 0.720] | 0.715 [0.637, 0.785] | 0.718 [0.615, 0.816] | 0.779 [0.704, 0.848] |
| Usual-meaning words | 21/142 | 7 (33%) | 9 (43%) | 21 | 0.603 [0.517, 0.692] | 0.644 [0.552, 0.730] | 0.810 [0.710, 0.897] | 0.921 [0.871, 0.965] |
| All | 229/761 | 93 (41%) | 142 (62%) | 130 | 0.616 [0.579, 0.656] | 0.617 [0.558, 0.681] | 0.689 [0.632, 0.750] | 0.848 [0.812, 0.881] |
| DeepSeek Flash (V4/4.1) | GPT-6-astra | |||||
|---|---|---|---|---|---|---|
| Corpus | Term class | Error | AUROC | Error | AUROC | |
| Synthetic | Redefined common words | 156 | 0.442 | 0.803 | 0.365 | 0.777 |
| Synthetic | Invented words | 158 | 0.253 | 0.779 | 0.127 | 0.720 |
| Synthetic | Usual-meaning words | 142 | 0.148 | 0.921 | 0.042 | 0.846 |
| Glossary term | 851 | 0.343 | 0.781 | 0.148 | 0.659 | |
| Control word | 540 | 0.235 | 0.842 | 0.143 | 0.815 | |
| Setting | Repair AUROC | Repair AUROC | Error AUROC difference , 95% CI | |
|---|---|---|---|---|
| Synthetic / DeepSeek V4 Flash | 456 | 0.833 | 0.810 | [-0.168, -0.046] |
| Synthetic / GPT-6-astra | 456 | 0.797 | 0.716 | [-0.259, -0.066] |
| CUAD library / GPT-5.6-terra | 1500 | 0.760 | 0.595 | [-0.395, -0.302] |
| Reddit / DeepSeek V4.1 Flash | 1391 | 0.744 | 0.709 | [-0.323, -0.244] |
| Reddit / GPT-6-astra | 1391 | 0.725 | 0.621 | [-0.364, -0.213] |
| DeFi / DeepSeek V4.1 Flash | 379 | 0.769 | 0.686 | [-0.445, -0.251] |
| Setting | error AUROC | Mean | Mean | Within- -decile AUROC |
|---|---|---|---|---|
| Synthetic / DeepSeek V4 Flash | 0.745 | .804 / .924 | .496 / .246 | .485 |
| Synthetic / GPT-6-astra | 0.656 | .751 / .920 | .450 / .287 | .431 |
| CUAD / GPT-5.6-terra | 0.464 | .407 / .714 | .139 / .141 | .398 |
| Reddit / DeepSeek V4.1 Flash | 0.520 | .440 / .675 | .163 / .131 | .409 |
| Reddit / GPT-6-astra | 0.429 | .405 / .639 | .105 / .147 | .359 |
| DeFi / DeepSeek V4.1 Flash | 0.458 | .410 / .670 | .116 / .130 | .411 |
| Term score | Uses questions? | Terms (shifted / consistent) | Class AUROC |
|---|---|---|---|
| No-context definition distance | No | 66 (38 / 28) | 0.946 |
| Domain-prior definition distance | No | 157 (118 / 39) | 0.977 |
| Model self-report | No | 79 (40 / 39) | 0.984 |
| Drafted-local/prior answer contrast | Yes | 78 (40 / 38) | 0.917 |
| Mean | Yes | 78 (40 / 38) | 0.783 |
| Mean | Yes | 78 (40 / 38) | 0.817 |
| Corpus | No-context distance [95% interval] | Domain-prior distance | Self-report |
|---|---|---|---|
| Synthetic | (65) | (186) | (118) |
| (211) | (217) | (217) | |
| DeFi | (57) | (64) | (64) |
| Setting | Bare failures | Other-item ( ) | ||
|---|---|---|---|---|
| Synthetic / DeepSeek V4 Flash | 130 | 0.445 | 0.983 | 0.914 (130) |
| Synthetic / GPT-6-astra | 83 | 0.421 | 0.962 | 0.976 (83) |
| CUAD / GPT-5.6-terra | 320 | 0.462 | 0.712 | 0.581 (190) |
| Reddit / DeepSeek V4.1 Flash | 419 | 0.453 | 0.858 | 0.824 (418) |
| Reddit / GPT-6-astra | 203 | 0.554 | 0.812 | 0.864 (203) |
| DeFi / DeepSeek V4.1 Flash | 107 | 0.501 | 0.863 | 0.850 (107) |
| Corpus | Bare | Source glossG | Drafted glossC | Prior glossM | |
|---|---|---|---|---|---|
| Synthetic | 761 | .699 | .966 (209/6) | .949 (196/6) | .708 (86/79) |
| Reddit glossary | 851 | .657 | .932 (243/9) | .790 (171/58) | .616 (82/117) |
| DeFi source-entry | 250 | .672 | .940 (70/3) | .760 (42/20) | .740 (40/23) |
| Corpus | Cue | Shuffle | No-cue | No-cue errors | No-cue AUROC: / repair- | |
|---|---|---|---|---|---|---|
| Synthetic | 456 | .686 | .067 | 143 | .308 .091 | .879 / .950 |
| 1,391 | .185 | .032 | 1,133 | .296 .210 | .810 / .844 | |
| DeFi | 379 | .274 | .086 | 275 | .284 .236 | .803 / .791 |
| Workload | Retrieval | Selected notes | All matched | Full glossary | |
|---|---|---|---|---|---|
| CUAD | 1,200 | 0.821 | 0.871 | 0.882 | 0.876 |
| Reddit / all | 1,391 | 0.866 | 0.925 | n/a | 0.907 |
| Reddit / glossary | 851 | 0.839 | 0.941 | n/a | 0.907 |
| Reddit / control | 540 | 0.907 | 0.898 | n/a | 0.907 |
| Corpus / tier / subset | Gain, pp [95% CI] | Repair/harm | Selected/full units | |
|---|---|---|---|---|
| CUAD / BM25 / all | 1200 | 4.17 [2.24, 6.12] | 94 / 44 | 135.8 / 529.2 |
| CUAD / dense / all | 1200 | 5.33 [3.49, 7.35] | 102 / 38 | 79.1 / 529.2 |
| CUAD / hybrid / all | 1200 | 5.00 [3.34, 6.73] | 93 / 33 | 126.6 / 529.2 |
| Reddit / BM25 / all | 1391 | 6.69 [4.45, 9.04] | 112 / 19 | 39.4 / 2765.9 |
| Reddit / BM25 / glossary | 851 | 10.34 [7.01, 13.83] | 97 / 9 | 46.5 / 3285.2 |
| Reddit / BM25 / control | 540 | 0.93 [-0.89, 2.78] | 15 / 10 | 28.3 / 1947.5 |
| Task | Arm | Score | Difference [95% CI] | |
|---|---|---|---|---|
| CUAD F1 | 1661 | rag_full | 0.5270 | +0.0029 [-0.0037, +0.0100] |
| CUAD F1 | 1661 | rag_gated | 0.5249 | +0.0008 [-0.0051, +0.0073] |
| ContractNLI acc. | 3060 | note_gated | 0.8173 | +0.0059 [-0.0020, +0.0147] |
| ContractNLI acc. | 3060 | note_all | 0.8222 | +0.0108 [+0.0026, +0.0193] |
| Reddit / GPT | 178 | note_gated | 0.5169 | +0.0169 [-0.0382, +0.0566] |
| Reddit / GPT | 178 | note_all | 0.5281 | +0.0281 [-0.0715, +0.0909] |
| Definition text | A direct: bare failures / repairs, AUROC [95% CI] | A to B: bare failures / repairs, AUROC [95% CI] |
|---|---|---|
| Drafted gloss | 144 / 85, 0.833 [0.773, 0.882] | 147 / 85, 0.811 [0.732, 0.893] |
| Community source entry | 144 / 116, 0.665 [0.539, 0.788] | 147 / 126, 0.600 [0.475, 0.717] |
| Model | Hugging Face commit (first 12 characters) |
|---|---|
| Qwen2.5-0.5B | 060db6499f32 |
| Qwen2.5-1.5B | 8faed761d45a |
| Qwen2.5-3B | 3aab1f1954e9 |
| Qwen2.5-7B | d14972939875 |
| Qwen2.5-14B | 97e1e76335b7 |
| Qwen2.5-14B-Instruct | cf98f3b3bbb4 |
Knowledge Conflicts in Language Models
Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.
See all 153 papers