Fine-Tuning Language Models to Know What They Know
Organizations: The University of Texas at Austin · Cognizant AI Lab
Abstract
Evaluating true metacognition in Large Language Models (LLMs) is difficult due to biases and heuristics. This paper presents a framework to measure and enhance LLM metacognition while controlling for these biases. A measurement method using the metric is established to isolate metacognitive ability. The Evolution Strategy for Metacognitive Alignment (ESMA) is proposed, demonstrating robust generalization across unseen datasets, languages, and newly acquired knowledge. Gradual parameter updates reveal that these improvements are driven by a sparse set of parameters, opening a potential avenue for targeted metacognitive optimization.
Figures & tables
| Model | Raw Alignment | Accuracy | Yes Ratio | YFR | NFR | |
|---|---|---|---|---|---|---|
| OpenAI GPT 5.2 | 0.94 | 84.66% | 85.24% | 92.59% | 12.25% | 53.95% |
| Gemini 3 Flash | 0.68 | 90.71% | 92.68% | 96.34% | 6.72% | 76.98% |
| Claude Sonnet 4.5 | 0.95 | 81.87% | 93.19% | 82.20% | 4.35% | 81.81% |
| Qwen2.5 1.5B | 0.20 | 53.30% | 42.86% | 53.81% | 53.56% | 38.69% |
| Qwen2.5 1.5B ESMA | 0.93 | 68.86% | 41.86% | 37.89% | 35.86% | 28.26% |
| Qwen2.5 3B | 0.29 | 62.70% | 35.67% | 18.22% | 54.46% | 33.47% |
| Model | Dual-question Accuracy | IDK Accuracy | IDK Alignment | All Alignment |
|---|---|---|---|---|
| Qwen2.5 1.5B | 42.86% | 38.21% | 52.34% | 31.45% |
| Qwen2.5 1.5B ESMA | 41.86% | 35.54% | 64.63% | 45.94% |
| Qwen2.5 3B | 35.67% | 35.38% | 59.88% | 28.87% |
| Qwen2.5 3B ESMA | 51.20% | 40.88% | 78.07% | 59.71% |
| Qwen2.5 7B | 50.43% | 49.53% | 60.05% | 34.93% |
| Qwen2.5 7B ESMA | 60.71% | 56.89% | 66.77% | 55.49% |
| Model | Raw Al. | Acc. | |
|---|---|---|---|
| Original | 0.23 | 76.18% | 1.45% |
| + FictionalQA | 0.20 | 53.92% | 51.64% |
| + ESMA | 0.65 | 48.97% | 51.09% |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | AP | Raw Alignment | Accuracy | |
|---|---|---|---|---|
| Qwen2.5 1.5B Non-Instruct | 14.70% | 0.44 | 60.38% | 42.99% |
| Qwen2.5 1.5B R-Tuning | 92.82% | 78.48% | 21.19% | |
| Qwen2.5 1.5B Instruct | – | 0.20 | 53.30% | 42.86% |
| Qwen2.5 1.5B ESMA | – | 0.93 | 68.86% | 41.86% |
| Size | Model | ECE | AUROC | Raw Alignment | Accuracy | |
|---|---|---|---|---|---|---|
| 1.5B | Original | 80.4% | 0.20 | 67.15% | 53.30% | 42.86% |
| CF | 7.8% | 0.43 | 69.88% | 60.32% | 41.77% | |
| ESMA | – | 0.93 | 75.07% | 68.86% | 41.86% | |
| 3B | Original | 73.3% | 0.29 | 60.10% | 62.70% | 35.67% |
| CF | 2.1% | 0.41 | 63.98% | 63.21% | 36.95% | |
| ESMA | – | 1.02 | 75.49% | 69.59% | 51.20% |
| Model | Raw Alignment | Accuracy | |
|---|---|---|---|
| Original | 0.20 | 53.30% | 42.86% |
| ESMA | 0.93 | 68.86% | 41.86% |
| Direct Correct | 0.24 | 54.67% | 44.17% |
| Meta Alignment | 0.65 | 67.09% | 35.69% |
| Size | Model | FreebaseQA | NQ Open | WebQuestions | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Raw Al. | Acc. | Raw Al. | Acc. | Raw Al. | Acc. | |||||
| 1.5B | Original | 0.30 | 54.55% | 42.57% | 0.19 | 54.46% | 19.47% | 0.39 | 60.09% | 34.15% |
| ESMA | 1.12 | 72.22% | 40.32% | 0.79 | 74.21% | 17.95% | 0.66 | 59.30% | 32.19% | |
| 3B | Original | 0.27 | 62.49% | 34.46% | 0.34 | 82.77% | 11.86% | 0.19 | 64.37% | 20.77% |
| ESMA | 1.17 | 71.92% | 50.28% | 0.84 | 62.44% | 22.60% | 0.60 | 55.86% | 37.11% | |
| 7B | Original | 0.73 | 64.19% | 49.40% | 0.53 | 63.46% | 21.91% | 0.50 | 55.95% | 36.71% |
| Model | Chinese | Korean | Spanish | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Raw Al. | Acc. | Raw Al. | Acc. | Raw Al. | Acc. | ||||
| Original | 0.08 | 13.09% | 6.78% | 0.27 | 73.43% | 2.83% | 0.49 | 23.92% | 9.99% |
| ESMA | 0.29 | 8.41% | 5.89% | 0.42 | 63.85% | 1.97% | 0.63 | 38.84% | 8.23% |
| Model | Raw Alignment | Accuracy | Yes Ratio | YFR | NFR | |
|---|---|---|---|---|---|---|
| Qwen2.5 1.5B | 0.20 | 53.30% | 42.86% | 53.81% | 53.56% | 38.69% |
| Qwen2.5 1.5B SFT | 0.40 | 62.02% | 38.82% | 22.45% | 48.13% | 35.04% |
| Qwen2.5 1.5B PPO | 0.31 | 57.32% | 42.87% | 40.21% | 49.76% | 37.91% |
| Qwen2.5 1.5B GRPO | 0.70 | 64.76% | 42.81% | 36.18% | 39.55% | 32.80% |
| Qwen2.5 1.5B ESMA | 0.93 | 68.86% | 41.86% | 37.89% | 35.86% | 28.26% |
| Model | Raw Alignment | Accuracy | |
|---|---|---|---|
| Qwen2.5 3B Original | 0.29 | 62.70% | 35.67% |
| Qwen2.5 3B PPO | 0.38 | 63.74% | 35.24% |
| Qwen2.5 3B GRPO | 0.87 | 70.47% | 35.01% |
| Qwen2.5 3B ESMA | 1.02 | 69.59% | 51.20% |
| Size | Model | Dual-question Acc. | IDK Acc. | IDK Al. | All Al. |
|---|---|---|---|---|---|
| 1.5B | Original | 42.86% | 38.21% | 52.34% | 31.45% |
| PPO | 42.87% | 38.27% | 52.23% | 28.52% | |
| GRPO | 42.81% | 37.57% | 55.11% | 33.82% | |
| ESMA | 41.86% | 35.54% | 64.63% | 45.94% | |
| 3B | Original | 35.67% | 35.38% | 59.88% | 28.87% |
| PPO | 35.24% | 34.98% | 59.93% | 30.76% |
| Size | Model | FreebaseQA | NQ Open | WebQuestions | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Raw Al. | Acc. | Raw Al. | Acc. | Raw Al. | Acc. | |||||
| 1.5B | Original | 0.30 | 54.55% | 42.57% | 0.19 | 54.46% | 19.47% | 0.39 | 60.09% | 34.15% |
| PPO | 0.36 | 58.08% | 42.62% | 0.31 | 60.91% | 19.20% | 0.32 | 60.78% | 33.96% | |
| GRPO | 0.92 | 68.52% | 42.89% | 0.57 | 69.42% | 19.34% | 0.54 | 60.09% | 34.60% | |
| ESMA | 1.12 | 72.22% | 40.32% | 0.79 | 74.21% | 17.95% | 0.66 | 59.30% | 32.19% | |
| 3B | Original | 0.27 | 62.49% | 34.46% | 0.34 | 82.77% | 11.86% | 0.19 | 64.37% | 20.77% |
| Size | Model | Chinese | Korean | Spanish | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Raw Al. | Acc. | Raw Al. | Acc. | Raw Al. | Acc. | |||||
| 1.5B | Original | 0.08 | 13.09% | 6.78% | 0.27 | 73.43% | 2.83% | 0.49 | 23.92% | 9.99% |
| PPO | 0.08 | 12.72% | 6.77% | 0.22 | 77.51% | 2.80% | 0.45 | 22.97% | 9.93% | |
| GRPO | 0.15 | 11.86% | 6.73% | 0.32 | 74.65% | 2.75% | 0.45 | 22.45% | 10.05% | |
| ESMA | 0.29 | 8.41% | 5.89% | 0.42 | 63.85% | 1.97% | 0.63 | 38.84% | 8.23% | |
| 3B | Original | 0.50 | 27.82% | 4.73% | 0.26 | 95.11% | 1.77% | 0.44 | 39.01% | 4.83% |
| Method | Elapsed Time | Relative Compute |
|---|---|---|
| ESMA | 3 h 47 min 39 s | 1.00 |
| PPO | 6 h 10 min 55 s | 1.63 |
| GRPO | 16 h 15 min 4 s | 4.28 |
| Size | Model | Raw Al. | Acc. | |
|---|---|---|---|---|
| 1.5B | Original | 0.08 | 41.72% | 75.26% |
| ESMA | 0.22 | 44.45% | 73.63% | |
| 3B | Original | 0.25 | 43.00% | 78.67% |
| ESMA | 0.39 | 76.02% | 80.46% | |
| 7B | Original | 0.19 | 65.61% | 78.24% |
| ESMA | 0.53 | 82.51% | 87.12% |
| Family | Model | Raw Al. | Acc. | AUROC | IDK Acc. | IDK Al. | All Al. | |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5 | Original | 0.90 | 56.97% | 68.96% | 71.26% | 54.51% | 84.80% | 46.57% |
| ESMA | 0.95 | 78.00% | 76.18% | 77.12% | 68.16% | 83.10% | 71.95% | |
| Qwen1.5 | Original | 0.21 | 51.62% | 46.81% | 64.92% | 48.80% | 71.23% | 39.24% |
| ESMA | 0.88 | 68.03% | 59.11% | 73.73% | 49.44% | 75.09% | 55.60% |
| Model | Metric | 0% | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3B | 0.29 | 0.86 | 0.94 | 0.99 | 1.04 | 1.03 | 1.03 | 1.03 | 1.03 | 1.03 | 1.02 | |
| Raw Al. | 62.70% | 66.28% | 68.15% | 69.06% | 69.85% | 69.68% | 69.70% | 69.61% | 69.72% | 69.68% | 69.59% | |
| 7B | 0.64 | 0.89 | 0.91 | 0.93 | 0.95 | 0.94 | 0.95 | 0.95 | 0.93 | 0.95 | 0.94 | |
| Raw Al. | 61.65% | 65.41% | 68.09% | 69.01% | 69.53% | 69.36% | 69.78% | 69.93% | 69.70% | 70.03% | 69.86% |
Metacognition in Language Models
Metacognition in language models studies how models assess their knowledge and reasoning, express uncertainty, and regulate their own actions.
See all 136 papers