Analogical reasoning is a hallmark of human intelligence, enabling us to solve new problems by transferring knowledge from one situation to another. Yet, developing artificial intelligence systems capable of robust human-like analogical reasoning has proven difficult. In this work, we train transformers using Meta-Learning for Compositionality (MLC) on an analogical reasoning task (letter-string analogies) and assess their generalization capabilities. We find that letter-string analogies become learnable when guiding the models to attend to the most informative problem elements, induced by including copy tasks in the training data. Furthermore, generalization to new alphabets improves when models are trained with more heterogeneous datasets. For the best training run, our 3-layer encoder-decoder model performs on par with frontier models on our letter-string analogy datasets. The MLC approach also enables some generalization to compositions of trained transformations, but not to completely novel transformations. To understand how the model solves the analogies, we identify an algorithm for one analogy task that approximates the model's computations. We verify this using interpretability analyses and show that head-level attention weights causally affect model output, allowing steering between different analogy tasks. We discuss the implications of our findings for generalization capabilities of larger models and the parallels to human analogical reasoning.
Figures & tables
Figure 1: A forward pass through the model, with an example letter-string analogy task.
Seen Transformations in Training
Transformation
Example
Extend (Ext)
a b c → a b c d
Successor (Succ)
a b c → a b d
Predecessor (Pred)
b c d → a c d
Remove Redundant (RmRed)
a b b c → a b c
Fix
a b w d → a b c d
Table 1: The full list of transformations used in this study. Transformation examples are shown for the standard alphabet only. ‘+’ is used as a shorthand to describe successive application of multiple transformations. The left side contains transformations that the models are trained on; the right column contains transformations that models hadn’t seen during training (compositional and novel).
Copy Tasks
Num. Seen Alphabets
Num. Examples
✗
20
1
✓
20
1
✓
40
1
✓
60
1
✓
80
1
✓
100
1
Table 2: Overview of all the dataset variants used for training.
Figure 4
Figure 3: Increasing the number of seen alphabets in training leads to better generalization to new alphabets (red and blue lines converge). Bands indicate 95% bootstrapped confidence intervals with 10 replications.
Figure 4: Averaged accuracies split by transformation with the best performing models with 10 replication runs (trained on a dataset with 200 permuted alphabets with copy tasks).
Figure 5: Models trained without copy tasks tend to show no information flow between example input and output (i.e., near-zero attention weights); models trained with copy tasks do.
Figure 6: Illustration of the initialization step (where role assignment happens) in the model. The upper part shows that the representations separate more cleanly after the first layer of the encoder. This is likely induced by the attention heads in the lower part of the figure.
Figure 7: Results from our attention pattern patching . A single attention head in the second encoder layer implements the matching operation of the algorithm. We demonstrate the effects of this head are causal , by replacing the standard attention matrix Apred with Asucc produced by a successor task.
Figure 8: The Compute mapping and Apply mapping steps of the algorithm. The encoder builds an invariant representation for the predecessor transformation and stores it in the first letter of the example output (left). This representation is then used in the decoder cross-attention (layer 2) to move attention from letter ’b’ (layer 1) to letter ’a’ (layer 3) of the decoder.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
copy
num. seen
batching method
accuracy seen transform. (%)
accuracy new transform. (%)
tasks
alphabets
seen alphabets
new alphabets
seen alphabets
new alphabets
✗
20
alphabet
56.3 [55.5, 57.5]
27.8 [27.8, 27.9]
10.2 [9.6, 10.9]
3.6 [3.2, 4.1]
random
65.3 [55.9, 79.4]
27.7 [26.3, 29.9]
9.4 [6.9, 12.5]
3.2 [2.3, 4.2]
transform.
64.5 [63.1, 65.6]
23.3 [23.0, 23.8]
13.6 [11.9, 15.7]
3.2 [2.8, 3.6]
transform. & alph.
57.6 [55.2, 59.8]
27.2 [26.5, 27.9]
7.6 [6.0, 9.3]
2.7 [2.1, 3.5]
✓
20
alphabet
90.2 [89.7, 90.7]
32.4 [31.2, 33.3]
10.0 [9.6, 10.5]
3.8 [3.5, 4.1]
Appendix
Table 4: Comparison of different batching methods, including copy tasks in the training set, and increasing the number of seen alphabets during training. Accuracies are averaged over five replication runs (highest averages per column in bold). Numbers in square brackets signify 95% bootstrapped confidence intervals.
Figure 9: Proportion of runs that develop a matching head split by training with and without copy tasks (datasets with 20 seen alphabets).
Transformation
Outcome
Prop. (%)
predecessor
applied succ
76.4
correct
14.9
other incorrect
8.7
successor
applied pred
85.1
correct
14.9
Appendix
Table 5: Performance of one of the models trained without copy tasks on predecessor- and successor tasks. For almost all errors, the model applies the other transformation, suggesting that the model cannot distinguish between the two transformations.
Figure 10: The best performing model forms distinct attention patterns for the example of each transformation type in the second encoder layer. For new transformations, (c) denotes compositional ones and (n) denotes novel ones. Where applicable, positions at which the transformation was applied were held constant across tasks of the same transformation.
Figure 11: Validation accuracy declines when providing the model with more examples to solve each task.
examples
Accuracy (%)
1
62.5
2
53.7
3
36.2
4
35.2
5
37.1
Appendix
Table 6: Overall accuracies of models trained with increasing number of examples per task.
Figure 12: Elimination step of the model where token representations that attend to the EOS token (left) are subsequently wiped out (right).
Figure 13: Comparison between matching step and human steps in analogical reasoning.
Figure 14: Frontier LLM performance by Transformation Type compared to our MLC trained models.
Figure 15: Frontier LLM performance when given examples compared to our MLC copy 200 trained model.
Figure 16: We observed clear parallels between our RASP-L implementation (top) and some of the attention heads in our MLC models (bottom). Most striking are the attention heads that correspond to the Arrow-selector (middle) and the Same-as selector (right).
Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoning, where a model transfers an attribute between entities that share known properties, and study when such transfer can emerge from training. To make the problem analytically tractable, we study a minimal transformer-style abstraction that isolates how learned representations support analogical reasoning. Within this setting, we prove three key results. First, joint training on similarity and attribution premises enables analogical reasoning through aligned representations. Second, sequential training succeeds only when similarity structure is learned before specific attributes, revealing a curriculum asymmetry. Third, in our stylized setting, two-hop reasoning (a→b,b→c⇒a→c) can be viewed as analogical reasoning with identity bridges (b=b), which appear explicitly in training data. Together, these results reveal a unified mechanism: entities with shared properties become aligned in representation space, enabling property transfer through feature resemblance. Experiments with architectures up to 8B parameters show qualitative agreement with the theory and suggest that representational geometry plays an important role in analogical reasoning beyond the stylized model.
Ruichen Xu, Wenjing Yan, Ying-Jun Angela Zhang
Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong.
Identifying and exploiting common features across domains is at the heart of the human ability to make analogies, and is believed to be crucial for the ability to continually learn. To do this successfully, general and flexible computational strategies must be developed. While the extent to which Transformer neural network models can perform compositional reasoning has been the subject of intensive recent investigation, little work has been done to systematically understand how well these models can leverage their representations to learn new, related experiences. To address this gap, we expand the previously developed Learning Equality and Group Operations (LEGO) framework to a continual learning (CL) setting ("continual LEGO"). Using this continual LEGO experimental paradigm, we study the capability of feedforward and recurrent Transformer models to perform CL. We find that BERT, a canonical feedforward Transformer model, learns shortcut solutions that limits its ability to generalize and prevents strong forward transfer to new experiences. In contrast, we find evidence supporting the hypothesis that ALBERT, a recurrent version of BERT, learns a For loop-esque solution, which leads to better CL performance. When applying BERT and ALBERT models to a CL setting that requires composition across experiences, we find that both model families fail. Our investigation suggests that ALBERT models can have their performance drop rescued by use of training strategies that combine data across experiences, but this is not true for BERT models, where a detrimental shortcut solution becomes entrenched with initial training. Our results demonstrate that the recurrent ALBERT model may have an inductive bias better suited for CL and motivate future investigation of the interplay between Transformer architecture and computational solutions that emerge in modern models and tasks.
William T. Redman, Erik C. Johnson, Brian Robinson
Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.