Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models
Organizations: Mahindra University · The University of Texas at Austin
Abstract
When a language model accepts a false correction in a factual discussion, its next answer can depend on how the conversation continues. We introduce Knowledge Answer Divergence (KAD), an event-conditioned evaluation protocol that distinguishes initial false-answer adoption from subsequent persistence, recovery, and abstention. KAD freezes the conversation at the first explicit adoption of a designated false target and branches independent probes from that exact history. This enables matched comparisons of post-adoption behavior and history-management interventions that a single susceptibility score cannot capture. We evaluate eight local checkpoint conditions and six hosted endpoints, alongside an audited 300-question rerun. In corrected Llama 3.2 Instruct 3B controls, a neutral-history branch yields 42/46 correct responses, compared with 16/46 under full history. A matched cue-placement control changes false adoption by 27.3 percentage points in Qwen 3.5 4B. Twelve independently authored pressure templates demonstrate substantial sensitivity to wording. Together, these findings demonstrate the utility of KAD for evaluating factual reliability beyond initial susceptibility, enabling systematic comparisons of false-answer persistence, recovery conditions and conversational interventions.
Figures & tables
| Model | Eligible | Collapse | Retained | Fresh | Held-out |
| Qwen 3.5 2B | 97 (6.5) | 87 (89.7) | 47 (54.0) | 83 (95.4) | 70 (80.5) |
| Qwen 3.5 4B | 241 (16.1) | 199 (82.6) | 185 (93.0) | 194 (97.5) | 129 (64.8) |
| Qwen 3.5 9B | 287 (19.1) | 159 (55.4) | 143 (89.9) | 151 (95.0) | 111 (69.8) |
| Llama 3.2 Instruct 1B | 97 (6.5) | 93 (95.9) | 10 (10.8) | 73 (78.5) | 47 (50.5) |
| Llama 3.2 Instruct 3B | 313 (20.9) | 300 (95.8) | 24 (8.0) | 267 (89.0) | 118 (39.3) |
| R1-Distill-Qwen 1.5B | 22 (1.5) | 20 (90.9) | 9 (45.0) | 17 (85.0) | 15 (75.0) |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Events | D/L false | S/A false | T/TE correct | F/FE correct | E/H correct |
| R1-0528-Qwen3 8B | 18 | 16/18 | 3/4 | 18/18 | 3/12 | 17/17 |
| Qwen 3.5 4B | 45 | 28/29 | 3/4 | 45/44 | 2/10 | 42/40 |
| Llama 3.2 Instruct 3B | 52 | 46/38 | 37/26 | 47/52 | 2/11 | 49/44 |
| Source | Questions | Share | Source split |
| PopQA | 500 | 33.3% | train snapshot |
| EntityQuestions | 400 | 26.7% | original entity fields |
| TriviaQA | 350 | 23.3% | rc.nocontext validation |
| NQ-Open | 250 | 16.7% | validation |
| Total | 1,500 | 100.0% | frozen benchmark |
| Study | Related measurement | KAD’s operational focus |
| FARM ( Xu et al., 2024 ) | Persuasion of initially correct factual answers and final belief checking. | Conditions continuation tests on observed first adoption of a designated false target. |
| SYCON ( Hong et al., 2025 ) | First-flip timing and repeated stance changes under user pressure. | Compares independent continuations from the same saved adoption history. |
| TRUTH DECAY ( Liu et al., 2025 ) | Answer changes under iterative feedback and rationale-based pressure. | Separates false adoption from later outcomes under specified history edits. |
| Reset Is Not Recovery ( Shnaidman, 2026 ) | Paired multiple-choice recovery contexts and option likelihoods. | Uses generated answers, a two-wording baseline gate and an observed-adoption branch point. |
| Model | Eligible | Collapse | Retained | Fresh | Held-out |
| Qwen 3.5 2B | 64 (21.3) | 59 (92.2) | 38 (64.4) | 57 (96.6) | 52 (88.1) |
| Qwen 3.5 4B | 110 (36.7) | 84 (76.4) | 82 (97.6) | 82 (97.6) | 61 (72.6) |
| Qwen 3.5 9B | 113 (37.7) | 46 (40.7) | 41 (89.1) | 43 (93.5) | 31 (67.4) |
| Llama 3.2 Instruct 1B | 12 (4.0) | 12 (100.0) | 5 (41.7) | 10 (83.3) | 10 (83.3) |
| Llama 3.2 Instruct 3B | 105 (35.0) | 98 (93.3) | 28 (28.6) | 84 (85.7) | 44 (44.9) |
| R1-Distill-Qwen 1.5B | 14 (4.7) | 14 (100.0) | 7 (50.0) | 13 (92.9) | 11 (78.6) |
| Model | Sequential | Early simultaneous | Difference [95% CI] | |
| Qwen 3.5 4B | 110 | 46.4% | 10.9% | [26.4, 44.5] |
| Llama 3.2 Instruct 3B | 105 | 65.7% | 56.2% | [2.9, 17.1] |
| R1-0528-Qwen3 8B | 80 | 71.2% | 18.8% | [41.2, 63.7] |
| Model | Mean effect | Template SD | Range | Template |
| Qwen 3.5 4B | pp | 10.1 pp | 0.0034 | |
| Llama 3.2 Instruct 3B | pp | 8.1 pp | 0.00049 | |
| R1-0528-Qwen3 8B | pp | 13.0 pp | 0.00049 |
| Model | General | Recognition-only | Reset-dependent | Delayed-only | Unrecovered |
| Qwen 3.5 2B | 47 | 0 | 39 | 1 | 0 |
| Qwen 3.5 4B | 185 | 1 | 13 | 0 | 0 |
| Qwen 3.5 9B | 143 | 2 | 14 | 0 | 0 |
| Llama 3.2 Instruct 1B | 10 | 10 | 65 | 5 | 3 |
| Llama 3.2 Instruct 3B | 24 | 20 | 245 | 0 | 11 |
| R1-Distill-Qwen 1.5B | 9 | 0 | 8 | 3 | 0 |
| Condition | Correct / cases | Recovery |
| Full collapse history | 68 / 126 | 54.0% |
| Neutral trigger replacement | 78 / 126 | 61.9% |
| Remove all pressure | 95 / 126 | 75.4% |
| Remove collapse trigger | 96 / 126 | 76.2% |
| Fresh context | 114 / 126 | 90.5% |
| Remove authority, authority collapses only | 4 / 5 | 80.0% |
| Model | Full | Neutral trigger | No pressure | Trigger removed | Fresh | |
| Qwen 3.5 2B | 10 | 5/10 | 8/10 | 10/10 | 10/10 | 10/10 |
| Qwen 3.5 4B | 31 | 27/31 | 30/31 | 29/31 | 30/31 | 29/31 |
| Qwen 3.5 9B | 23 | 21/23 | 23/23 | 23/23 | 23/23 | 21/23 |
| Llama 3.2 Instruct 1B | 16 | 8/16 | 8/16 | 10/16 | 13/16 | 12/16 |
| Llama 3.2 Instruct 3B | 46 | 7/46 | 9/46 | 23/46 | 20/46 | 42/46 |
| Panel | Without policy | With policy | Difference |
| Primary false persistence | 2/200 | 0/200 | pp |
| R1-0528-Qwen3 8B false persistence | 4/90 | 0/90 | pp |
| Primary clean correctness | 80/80 | 80/80 | 0.0 pp |
| R1-0528-Qwen3 8B clean correctness | 40/40 | 40/40 | 0.0 pp |
| Primary true-correction correctness | 163/200 | 0/200 | pp |
| Primary false-correction correctness | 0/200 | 5/200 | pp |
| Model | Baseline | Collapse | Retained | Fresh | |
| Llama 3.2 Instruct 3B, expanded | 41 | 3.70 | -4.16 | 1.07 | 3.50 |
| Qwen 3.5 4B, expanded | 20 | 2.10 | -1.51 | 0.57 | 2.14 |
| Llama 3.2 Instruct 3B | 27 | 4.04 | -3.82 | 1.33 | 3.85 |
| Qwen 3.5 2B | 13 | 1.78 | -2.77 | -0.56 | 1.83 |
| Qwen 3.5 4B | 14 | 2.76 | -1.35 | 0.98 | 2.84 |
| Qwen 3.5 9B | 15 | 2.88 | -1.50 | 0.42 | 2.77 |
| Model | Collapsed trajectories | High-conf. false | Raw [95% CI] | Margin-output agreement |
| Qwen 3.5 4B | 20 | 30/31 | 0.187 [ , 0.400] | 83.2% |
| Llama 3.2 Instruct 3B | 41 | 93/93 | 0.126 [0.008, 0.230] | 94.1% |
| Measure | All adjudicated | Manual clean |
| Baseline-eligibility F1 | 0.989 | 1.000 |
| False-target-adoption F1 | 0.992 | 1.000 |
| Retained-context F1 | 0.958 | 0.962 |
| Fresh-context F1 | 0.969 | 0.968 |
| Final-profile accuracy | 0.875 | 0.958 |
| Trajectory rows | 160 | 142 |
| Field | Agreement | AC1 | |
| Canonical answer correct | .993 | -.001 | .993 |
| Unambiguous single answer | .770 | -.038 | .743 |
| Factual and atomic | 1.000 | 1.000 | 1.000 |
| Temporally stable | .938 | -.027 | .934 |
| Baseline paraphrase equivalent | .997 | .000 | .997 |
| Held-out paraphrase equivalent | .987 | .000 | .987 |
| Panel | Outcome | Pass | Fail | Difference [95% CI] |
| Local original | Eligibility | 3008 | 1712 | [.037,.090] |
| Local original | Collapse eligible | 317 | 72 | [ ] |
| Local original | Retained recovery | 248 | 55 | [ ] |
| Local original | Fresh recovery | 248 | 55 | [ ] |
| Local original | Held-out recovery | 248 | 55 | [ ] |
| Hosted common | Eligibility | 1248 | 552 | [.110,.313] |
| Endpoint | Retained | Fresh | Held-out | |
| Gemma 4 26B-A4B | 6 | 6 | 5 | 3 |
| Gemma 4 31B | 6 | 5 | 4 | 3 |
| GPT-OSS 20B | 19 | 15 | 16 | 10 |
| GPT-OSS 120B | 13 | 11 | 10 | 7 |
| Qwen 3.6 27B | 16 | 15 | 16 | 9 |
| Qwen 3.8 27B | 27 | 27 | 24 | 25 |
| Study/model | Branch | Correct | False | |
|---|---|---|---|---|
| DS7 diagnostic/DS7b | baseline | 24 | 24 | 0 |
| DS7 diagnostic/DS7b | fresh | 24 | 0 | 9 |
| DS7 diagnostic/DS7b | retained | 24 | 24 | 0 |
| History/DS8b | Baseline exchange | 34 | 34 | 0 |
| History/DS8b | fresh | 34 | 32 | 0 |
| History/DS8b | full | 34 | 30 | 4 |
| Model/contrast | Difference | Low | High | Holm | |
|---|---|---|---|---|---|
| History/DS8b: baseline–fresh | 34 | 5.9 | 0.0 | 14.7 | 1 |
| History/DS8b: full–neutral | 34 | -8.8 | -20.6 | 2.9 | 1 |
| History/L3b: baseline–fresh | 46 | -19.6 | -34.8 | -6.5 | 0.112 |
| History/L3b: full–neutral | 46 | -56.5 | -69.6 | -43.5 | 1.79e-07 |
| History/Q4b: baseline–fresh | 31 | 6.5 | 0.0 | 16.1 | 1 |
| History/Q4b: full–neutral | 31 | -9.7 | -22.6 | 0.0 | 1 |
| Term | OR | Low | High | |
| Intercept | 10.9 | 1.95 | 60.4 | 0.0065 |
| full_history | 0.0551 | 0.0234 | 0.13 | 3.14e-11 |
| neutral_trigger_replacement | 0.0965 | 0.0424 | 0.22 | 2.62e-08 |
| remove_all_pressure | 0.251 | 0.117 | 0.537 | 0.000367 |
| remove_authority | 0.203 | 0.0276 | 1.49 | 0.117 |
| remove_trigger | 0.266 | 0.122 | 0.581 | 0.0009 |
| Axis | Category | Sample / 138 | Estimate / 756 |
| content | correct | 79 | 560.0 |
| content | designated-false | 24 | 48.5 |
| content | other | 9 | 19.5 |
| content | no-answer | 12 | 78.5 |
| content | ambiguous | 14 | 49.5 |
| commitment | direct | 97 | 578.0 |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 862 papers