Feeling of Knowing in Large Language Models
Organizations: City University of Hong Kong · Tencent Jarvis Lab · Xi’an Jiaotong University · Renmin University of China ·
Abstract
Post-training updates change what large language models (LLMs) can answer, but may leave their answerability judgments misaligned. We study this knowledge-judgment desynchronization through functional feeling of knowing (FoK), which supports selective generation, retrieval triggering, and model routing without implying subjective experience. We propose two trainable channels: a knowledge channel that changes answerability and a FoK channel that judges the current knowledge state. Three stages train them: on-policy supervised fine-tuning (OP-SFT) updates knowledge from corrected on-policy answers; knowledge-delta FoK training (K-FT) adapts judgments across intermediate knowledge states; and FoK-guided policy optimization (FGPO) trains the subsequent answer or explanation. Factual-QA experiments, cross-dataset evaluation, and three sequential updates show that explicit FoK alignment improves judgment quality on updated knowledge. MMLU and ARC-Easy separately assess capability preservation.
Figures & tables
| Base | K-SFT | J-SFT | K J SFT | |||||
|---|---|---|---|---|---|---|---|---|
| Model | Acc. | AUROC | Acc. | AUC | Acc. | AUC | Acc. | AUC |
| Qwen2.5-0.5B | 20% | 0.519 | +2 | -0.016 | -14 | +0.213 | +2 | +0.211 |
| Qwen2.5-1.5B | 35% | 0.529 | +6 | -0.035 | +2 | +0.255 | +5 | +0.248 |
| Qwen2.5-3B | 39% | 0.739 | +12 | -0.220 | +3 | +0.038 | +11 | +0.051 |
| Qwen2.5-7B | 53% | 0.724 | +7 | -0.042 | +2 | +0.088 | +7 | +0.100 |
| Qwen2.5-14B | 66% | 0.762 | +6 | +0.004 | +1 | +0.030 | +5 | +0.068 |
| Train | Test | General | ||||||
| Method | Acc. | AUROC | Bias | Acc. | AUROC | Bias | MMLU | ARC |
| Base | 0.680 | 0.702 | 0.381 | 0.672 | 0.688 | 0.402 | 0.687 | 0.811 |
| + Prob | 0.680 | 0.723 | 0.409 | 0.672 | 0.730 | 0.407 | 0.687 | 0.811 |
| + N-Prob | 0.680 | 0.723 | 0.409 | 0.672 | 0.730 | 0.407 | 0.687 | 0.811 |
| + EliCal | 0.637 | 0.808 | 0.263 | 0.622 | 0.810 | 0.254 | 0.709 | 0.504 |
| Vanilla K-SFT | 0.874 | 0.458 | 0.500 | 0.546 | 0.519 | 0.500 | 0.713 | 0.840 |
| Stage | Acc. | AUROC | Illu. | Om. | Bias |
|---|---|---|---|---|---|
| Base | 0.672 | 0.688 | 0.683 | 0.122 | 0.402 |
| Stage 1: + OP-SFT | 0.679 | 0.603 | 0.601 | 0.283 | 0.442 |
| Stage 2: + K -FT | 0.699 | 0.770 | 0.478 | 0.187 | 0.333 |
| Stage 3: + FGPO | 0.692 | 0.809 | 0.432 | 0.184 | 0.308 |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Backbone | Qwen2.5-7B-Instruct |
| Training framework | LLaMA Factory |
| Adapter type | LoRA, rank 16, alpha 32, dropout 0 |
| Parallelism | Multi-GPU DDP with DeepSpeed ZeRO-2 |
| Precision | bfloat16 |
| Vanilla K-SFT LR / epochs | / 5 |
| Response outcome | Base reward | ||
|---|---|---|---|
| 1 | yes | correct answer | |
| 1 | yes | wrong answer | |
| 1 | no | — | |
| 0 | no | valid explanation | |
| 0 | no | invalid explanation | |
| 0 | yes | correct answer |
| Asset | Use in this paper | License or terms |
| TriviaQA | Knowledge-update corpus and in-domain QA benchmark | Apache-2.0 ( source ) |
| MMLU | General capability benchmark | MIT License ( source ) |
| ARC-Easy | General capability benchmark | CC BY-SA 4.0 ( source ) |
| Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct | Main answer model and LLM judge | Apache-2.0 (Qwen 7B and 32B model cards) |
| Qwen3-8B | Thinking-mode backbone for the selective-prediction evaluation | Apache-2.0 (Qwen model card ) |
| LLaMA-3.1 and LLaMA-2 checkpoints | Supplementary FoK synchronization results in Appendix G | Meta Llama 3.1 Community License and Llama 2 Community License ( 3.1 , 2 ) |
| Base | K-SFT | J-SFT | K J SFT | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | EM | AUROC | EM | AUROC | Illu. | Om. | AUROC | EM | AUROC | Illu. |
| 0.5B | 4.2% | 0.523 | 16.9% | 0.502 | 0% | 99% | 0.801 | 13.1% | 0.763 | 3% |
| 1.5B | 30.9% | 0.527 | 35.5% | 0.500 | 66% | 48% | 0.824 | 34.8% | 0.815 | 28% |
| 3B | 35.4% | 0.724 | 45.7% | 0.519 | 24% | 80% | 0.830 | 45.4% | 0.822 | 40% |
| 7B | 47.2% | 0.735 | 55.8% | 0.592 | 28% | 51% | 0.855 | 55.1% | 0.833 | 82% |
| 14B | 59.8% | 0.756 | 65.2% | 0.767 | 73% | 9% | 0.850 | 66.1% | 0.814 | 85% |
| Model | Base AUROC | K-SFT AUROC | J-SFT AUROC | K J SFT AUROC | K J SFT EM | Note |
|---|---|---|---|---|---|---|
| Qwen2.5-7B | 0.735 | 0.592 | 0.855 | 0.833 | 55.1% | Reference |
| LLaMA-3.1-8B (LR ) | 0.646 | 0.503 | 0.786 | 0.836 | 8.6% | K J forgetting |
| LLaMA-3.1-8B (LR ) | 0.646 | 0.526 | 0.782 | 0.749 | 73.5% | LR corrected |
| LLaMA-2-7B | 0.520 | 0.500 | 0.500 | 0.500 | 63.5% | No FoK improvement |
| LLaMA-2-13B | 0.525 | 0.500 | 0.500 | 0.500 | 70.1% | No FoK improvement |
| Answer model | FoK estimator | Test Acc. | Test AUROC |
|---|---|---|---|
| Base | Direct judgment | 0.672 | 0.688 |
| Base | Prob | 0.672 | 0.730 |
| Base | EliCal | 0.622 | 0.810 |
| OP-SFT | Direct judgment | 0.679 | 0.603 |
| OP-SFT | Prob | 0.679 | 0.571 |
| OP-SFT | EliCal | 0.679 | 0.618 |
| New-shard | Test | (frozen) | (adaptive) | |||
|---|---|---|---|---|---|---|
| State | Acc. before after | Acc. | AUROC | Illusion | AUROC | Illusion |
| Base | — | 0.672 | — | — | — | — |
| Round 1 | 0.659 | 0.798 | 0.455 | 0.798 | 0.455 | |
| Round 2 | 0.644 | 0.776 | 0.688 | 0.800 | 0.202 | |
| Round 3 | 0.661 | 0.754 | 0.752 | 0.797 | 0.195 | |
| SA | NQ | WebQ | ||||
|---|---|---|---|---|---|---|
| System | Acc. | AUROC | Acc. | AUROC | Acc. | AUROC |
| Base | 0.282 | 0.768 | 0.376 | 0.665 | 0.527 | 0.597 |
| Base + EliCal | 0.282 | 0.702 | 0.376 | 0.735 | 0.527 | 0.636 |
| Vanilla K-SFT | 0.187 | 0.569 | 0.191 | 0.571 | 0.224 | 0.519 |
| OP-SFT + EliCal | 0.305 | 0.590 | 0.381 | 0.579 | 0.539 | 0.515 |
| Ours | 0.305 | 0.810 | 0.381 | 0.733 | 0.539 | 0.644 |
| System | AUROC | TPR | TNR | Answer Acc. |
|---|---|---|---|---|
| Base | 0.768 | 0.388 | 0.898 | 0.282 |
| Vanilla K-SFT | 0.569 | 0.302 | 0.760 | 0.187 |
| Ours | 0.810 | 0.765 | 0.701 | 0.305 |
| System | Held-out Acc. | FoK AUROC |
|---|---|---|
| Base | 0.319 | 0.629 |
| Vanilla K-SFT | 0.199 | 0.585 |
| Ours | 0.366 | 0.647 |
| Train Acc. | Train AUROC | Test Acc. | Test AUROC | |
|---|---|---|---|---|
| Before edit | 0.035 | 0.594 | 0.633 | 0.726 |
| After edit | 0.640 | 0.553 | 0.627 | 0.709 |
| Vanilla K-SFT | OP-SFT + K -FT | |||
| Seed | Acc. | AUROC | Acc. | AUROC |
| 42 | — | — | 0.699 | 0.770 |
| 43 | 0.522 | 0.635 | 0.665 | 0.770 |
| 44 | 0.522 | 0.627 | 0.670 | 0.790 |
| 0 | 0.520 | 0.619 | 0.665 | 0.778 |
| Mean SD | ||||
| State | Test Acc. | Test AUROC |
|---|---|---|
| Shared Stage-1+2 initialization | 0.699 | 0.770 |
| FGPO, seed 1 | 0.692 | 0.809 |
| FGPO, seed 43 | 0.677 | 0.810 |
| FGPO, seed 44 | 0.672 | 0.815 |
| FGPO mean SD |
| Answer consistency | Questions | Mean |
|---|---|---|
| 0–0.2 | 840 | 0.413 |
| 0.2–0.4 | 178 | 0.486 |
| 0.4–0.6 | 71 | 0.571 |
| 0.6–0.8 | 29 | 0.591 |
| 0.8–1.0 | 31 | 0.808 |
| Reasoning shown | AUROC | Mean- gap |
|---|---|---|
| Question only | 0.60 | 0.05 |
| 25% | 0.66 | 0.09 |
| 50% | 0.67 | 0.10 |
| 75% | 0.74 | 0.15 |
| Full reasoning | 0.91 | 0.32 |
| Format | Condition | Acc. | Acc. | ||
|---|---|---|---|---|---|
| — | None | 0.599 | — | 0.408 | — |
| Words | Unrelated | 0.184 | 0.424 | ||
| Words | Related | 0.674 | 0.478 | ||
| Sentences | Unrelated | 0.298 | 0.436 | ||
| Sentences | Correct related | 0.677 | 0.510 | ||
| Sentences | Incorrect related | 0.772 | 0.397 |
| Template | Peak layer | Sensitive range |
|---|---|---|
| Original | 20 | 16–22 |
| A: rephrased | 21 | 17–22 |
| B: third-person elicitation | 21 | 16–21 |
| C: numeric labels 1/2 | 21 | 18–23 |
| D: letter labels A/B | 21 | 18–21 |
| E: labels can/cannot | 20 | 18–20 |
| Strategy | Update Acc. | Held-out Acc. | GSM8K | Average |
|---|---|---|---|---|
| Base | 0.685 | 0.603 | 0.798 | 0.695 |
| Vanilla SFT | 0.992 | 0.452 | 0.368 | 0.604 |
| Reptile meta-update | 0.558 | 0.584 | 0.640 | 0.594 |
| SDFT | 0.529 | 0.565 | 0.367 | 0.487 |
| Unlikelihood on wrong span | 0.500 | 0.528 | — | — |
| One-step lookahead KL | 0.738 | 0.619 | 0.646 | 0.668 |
| Asset | Role | License or terms |
|---|---|---|
| SelfAware | Zero-shot QA | CC BY-SA 4.0 ( source ) |
| Natural Questions | Transfer and replication | CC BY-SA 3.0 ( data source ) |
| WebQuestions | Zero-shot QA | CC BY 4.0 ( source ) |
| MATH | Reasoning evaluation | MIT ( repository ) |
| GSM8K | Capability evaluation | MIT ( repository ) |
| MEMIT | Editing implementation | MIT ( repository ) |
| Model | Acc. (%) | AUROC | Illusion (%) | Omission (%) |
|---|---|---|---|---|
| Qwen2.5-0.5B | 20 | 0.519 | 28 | 85 |
| Qwen2.5-1.5B | 35 | 0.529 | 56 | 38 |
| Qwen2.5-3B | 39 | 0.739 | 8 | 68 |
| Qwen2.5-7B | 53 | 0.724 | 51 | 20 |
| Qwen2.5-14B | 66 | 0.762 | 50 | 19 |
| Qwen2.5-32B | 68 | 0.819 | 62 | 8 |
| K-SFT | J-SFT | K J SFT | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Acc. | AUC | Illu. | Om. | Acc. | AUC | Illu. | Om. | Acc. | AUC | Illu. | Om. |
| Qwen2.5-0.5B | +2 | -0.016 | -27 | +13 | -14 | +0.213 | -28 | +15 | +2 | +0.211 | -21 | -10 |
| Qwen2.5-1.5B | +6 | -0.035 | +6 | -2 | +2 | +0.255 | -28 | -6 | +5 | +0.248 | -21 | -12 |
| Qwen2.5-3B | +12 | -0.220 | +10 | +18 | +3 | +0.038 | +11 | -29 | +11 | +0.051 | +20 | -41 |
| Qwen2.5-7B | +7 | -0.042 | -15 | +19 | +2 | +0.088 | -9 | -4 | +7 | +0.100 | +6 | -12 |
| Qwen2.5-14B | +6 | +0.004 | +9 | -4 | +1 | +0.030 | +9 | -9 | +5 | +0.068 | +29 | -17 |
| Stage | Acc. | AUROC | Illu. | Om. | Bias |
|---|---|---|---|---|---|
| Base | 0.680 | 0.702 | 0.653 | 0.109 | 0.381 |
| Stage 1: + OP-SFT | 0.966 | 0.575 | 0.589 | 0.319 | 0.454 |
| Stage 2: + K -FT | 0.967 | 0.803 | 0.350 | 0.190 | 0.270 |
| Stage 3: + FGPO | 0.961 | 0.817 | 0.250 | 0.240 | 0.245 |
Metacognition in Language Models
Metacognition in language models studies how models assess their knowledge and reasoning, express uncertainty, and regulate their own actions.
See all 136 papers