Tracing Knowledge and Reasoning in LLMs
Organizations: University of Manchester · Shanghai University of Finance and Economics · University of Sheffield
Abstract
Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Domain | Question Type | Metric | # Examples |
|---|---|---|---|---|
| BioMixQA Soman et al. (2024) | Biomedical | Multiple-choice | Accuracy | 306 |
| MedMCQA Pal et al. (2022) | Biomedical | Multiple-choice | Accuracy | 4,183 |
| MRI-MCQA Sedano (2025) | Biomedical | Multiple-choice | Accuracy | 563 |
| PubMedQA Jin et al. (2019) | Biomedical | Yes/No | Accuracy | 1,000 |
| SciQ Welbl et al. (2017) | Science | Multiple-choice | Accuracy | 1,000 |
| QASC Khot et al. (2020) | Science | Multiple-choice | Accuracy | 926 |
| Models | Weights | Model Link | License | Release Time |
|---|---|---|---|---|
| Llama3-8B | Dense | meta-llama/Meta-Llama-3-8B-Instruct | Llama 3 Community | Apr 2024 |
| Llama3.1-8B | Dense | meta-llama/Llama-3.1-8B-Instruct | Llama 3.1 Community | Jul 2024 |
| Qwen2.5-3B | Dense | Qwen/Qwen2.5-3B-Instruct | Apache 2.0 | Sep 2024 |
| Qwen2.5-7B | Dense | Qwen/Qwen2.5-7B-Instruct | Apache 2.0 | Sep 2024 |
| Qwen2.5-14B | Dense | Qwen/Qwen2.5-14B-Instruct | Apache 2.0 | Sep 2024 |
| Qwen3-8B | Dense | Qwen/Qwen3-8B | Apache 2.0 | Apr 2025 |
| Metric | Lower | Middle | Upper | |||
|---|---|---|---|---|---|---|
| Attention Entropy | .001 | .42 | .055 | .14 | .001 | .45 |
| Activation Sparsity | .999 | .27 | .999 | .07 | .205 | .13 |
| Capability | Median change | Models improved | p-value | Effect size |
|---|---|---|---|---|
| Reasoning | +.036 | 14/16 | .008 | +.74 |
| Knowledge | +.001 | 9/16 | .597 | +.16 |
| Perturbation | Median accuracy drop | Significant comparisons |
|---|---|---|
| Nota -as-answer | .346 | 24/24 |
| Nota -as-distractor | .012 | 3/24 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,539 papers