cs.CLApr 4, 2026
SaveWhen Models Know More Than They Say: Probing Analogical Reasoning in LLMs
Organizations: Cambridge University · Northeastern University · Athenahealth · Johns Hopkins University
Abstract
Analogical reasoning is a core cognitive faculty essential for narrative understanding. While LLMs perform well when surface and structural cues align, they struggle in cases where an analogy is not apparent on the surface but requires latent information, suggesting limitations in abstraction and generalisation. In this paper we compare a model's probed representations with its prompted performance at detecting narrative analogies, revealing an asymmetry: for rhetorical analogies, probing significantly outperforms prompting in open-source models, while for narrative analogies, they achieve a similar (low) performance. This suggests that the relationship between internal representations and prompted behavior is task-dependent and may reflect limitations in how prompting accesses available information.
Figures & tables
Figure 1: Analogical reasoning is a higher-order capability that requires a combination of lower-level tasks such as entity detection or coreference resolution.
| Anchor | When I remember the challenges I went through when I was starting my business, I break into tears. But I do not regret a thing. I think that the most precious gold goes through the hottest furnace. It made me better. |
|---|---|
| Analogy | Once upon a time, in a small village, there lived a talented young presenter named Lily. She faced repeated challenges, but each obstacle made her stronger and more resilient, ultimately earning her respect and admiration. |
Table 1: Example narrative parallelism pair from the Analogy dataset
| Latin | English |
|---|---|
| satietas sitiret | satiety might thirst , |
| uirtus infirmaretur | strength might be weakened , |
| sanitas uulneraretur | health might be wounded , |
| uita moreretur | life might die . |
Table 2: Rhetorical Parallelism in Latin: The repeated syntactic and semantic inversion emphasizes paradoxical transformation.
Figure 2: MAP for narrative (left) and rhetorical (right) parallelism tasks across different classifier architectures and model variants on Llama-3.2-1B. Bars show MAP scores (mean ± standard deviation) for three classifier types: cosine similarity (Cosine), logistic regression (Logreg), and multi-layer perceptron (MLP), with separate bars for base and instruction-tuned (Instruct) model variants.
Figure 3: Individual layer performance vs. all-layers configuration on Llama-3.2-1B base model with MLP classifiers. Top panels show learned layer weights from ScalarMix (averaged across cross-validation folds), indicating the relative contribution of each layer when all layers are combined. Bottom panels show mean average precision (MAP) for each individual layer (bars with error bars showing standard deviation) and the all-layers performance (red dashed horizontal line).
Figure 4: MAP on prompted ranking across different models
| Task | Model | MAP | MRR | Pairwise Acc. |
|---|---|---|---|---|
| Narrative | Claude Opus | 0.8158 | 0.9127 | 0.8892 |
| GPT-5.2 | 0.8181 | 0.9126 | 0.8921 | |
| Llama-1B-Instruct | 0.3486 | 0.4487 | 0.4937 | |
| Llama-8B-Instruct | 0.2987 | 0.3540 | 0.4512 | |
| Rhetorical | Claude Opus | 0.9084 | 0.9077 | 0.9558 |
| GPT-5.2 | 0.9000 | 0.9009 | 0.9753 |
Table 3: Prompting experiments on narrative and rhetorical parallelism tasks. Results show that closed-source models (Claude Opus, GPT-5.2) substantially outperform open-source LLaMA models on both tasks, with particularly large gaps on rhetorical parallelism.
| Classifier | MAP (mean std) | |
|---|---|---|
| Cosine | ||
| Distance | ||
| Logreg | ||
| MLP | ||
| Full |
Table 4: MAP (mean ± std) for Rhetorical Task, Base Variant, 1B Model
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Variant | Classifier | MRR | MAP | Accuracy | |||
|---|---|---|---|---|---|---|---|---|
| Narrative | Base | Cosine | ||||||
| Logreg | ||||||||
| MLP | ||||||||
| Narrative | Instruct | Cosine | ||||||
| Logreg | ||||||||
| MLP | ||||||||
Table 5: MRR, MAP, and Accuracy (mean ± std) for different classifiers
Figure 5: Distribution of branch sizes based on the number of spans per parallel set
Figure 6: Distribution of proverb sizes based on the number of narratives per proverb
Table 6: Examples from the ARN dataset: Easies and hardest analogies
| Task | Provenance | # Unique Docs | Task Type |
|---|---|---|---|
| Event Det. | Lit-Bank | 100 documents | Span Classification |
| Entity Det. | Lit-Bank | 100 documents | Span Classification |
| Entity Coref. | Lit-Bank | 100 documents | Span Pair Classification |
| Quote Attr. | Lit-Bank | 100 documents | Span Pair Classification |
| Rhetorical Sym. | Augustinian Sermon Parallelism dataset | 80 sermons | Ranking |
| Narrative Sym. | Analogical Reasoning over Narratives dataset from ePiC stories | 872 narratives | Ranking |
Table 7: Dataset description by task. This table shows the provenance of each dataset alongside the number of unique source documents and task type.
Figure 7: 1/3/8B Llama F1 vs Lit-Bank Task (Pooling: Mean, Classifier: MLP)
Figure 8: 1/3/8B Llama F1 vs Lit-Bank Task (Pooling: Mean, Classifier: LogReg)
Figure 9: 1/3/8B Llama F1 vs Lit-Bank Task (Pooling: Max, Classifier: MLP)
Figure 10: 1/3/8B Llama F1 vs Lit-Bank Task (Pooling: Max, Classifier: LogReg)
Figure 11: Similarity Scores across 446 ARN Document Pairs (Normalized with Min-Max Scaling) Using Non-LLM-Based Methods
Figure 12: Similarity Scores across 564 ASP Span Pairs (Normalized with Min-Max Scaling) Using Non-LLM-Based Methods