Organizations: School of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, China. · Institute of Biomedical Engineering, Ningbo institute of materials technology and engineering, Chinese Academy of Sciences. · School of Electrical & Electronic Engineering, Nanyang Technological University, Nanyang Link, 639798 Singapore.
While visual reasoning for simple analogies has received significant attention, compositional visual relations (CVR) remain relatively unexplored due to their greater complexity. To solve CVR tasks, we propose Predictive Reasoning with Augmented Anomaly Contrastive Learning (PR-A2CL), \ie, to identify an outlier image given three other images that follow the same compositional rules. To address the challenge of modelling abundant compositional rules, an Augmented Anomaly Contrastive Learning is designed to distil discriminative and generalizable features by maximizing similarity among normal instances while minimizing similarity between normal and anomalous outliers. More importantly, a predict-and-verify paradigm is introduced for rule-based reasoning, in which a series of Predictive Anomaly Reasoning Blocks (PARBs) iteratively leverage features from three out of the four images to predict those of the remaining one. Throughout the subsequent verification stage, the PARBs progressively pinpoint the specific discrepancies attributable to the underlying rules. Experimental results on SVRT, CVR and MC2R datasets show that PR-A2CL significantly outperforms state-of-the-art reasoning models.
Figures & tables
Fig. 1: Left : A sample compositional rule in the CVR dataset. Right : Selecting an outlier from four images is converted to four predict-and-verify problems.
Fig. 2: Proposed PR-A 2 CL comprises two modules: 1) A Perception Module with Augmented Anomaly Contrastive Learning (A 2 CL), generating weak and strong augmented views, encoding them via ResNet-50, and hence improving feature discriminability and generalization. 2) A Predictive Anomaly Reasoning Module (PARM) built with stacked PARBs. Each block employs a predict-and-verify mechanism, which predicts the target from context features and verifies it against the target to infer latent compositional rules.
Model
20
50
100
200
500
1000
AUC
10000
ViT-Small [ 2 ]
24.7
24.8
24.9
26.6
27.6
29.8
26.4
40.0
IN-ViT-Small [ 2 ]
25.6
25.1
26.2
26.1
28.7
33.0
27.5
92.4
SCAR [ 41 ]
25.8
25.9
27.0
28.3
31.0
32.9
28.5
50.3
WReN [ 40 ]
26.0
28.2
27.9
29.7
33.0
39.7
30.8
70.3
SCL [ 29 ]
25.8
25.1
25.9
26.6
31.9
55.0
31.7
73.4
ResNet-50 [ 1 ]
29.8
31.5
32.4
33.4
41.6
54.0
37.1
96.0
TABLE I: State-of-the-art comparison on SVRT [ 36 ] under the same settings as [ 34 ] . Our method consistently and significantly outperforms all baselines across all training sample sizes.
Fig. 3: Comparison with DBCR [ 35 ] on SVRT [ 36 ] using 1k samples per task. PR-A 2 CL excels across most tasks, whereas DBCR struggles on several.
Model
Accuracy (%) for Using Different Numbers of Samples per Task for Joint/Individual Training
Para. (M)
GFLOPs
20
50
100
200
500
1000
AUC
10000
SCAR [ 41 ]
26.2/27.1
26.2/27.4
26.6/27.6
27.1/28.0
27.7/28.0
29.4/32.3
27.4/28.4
42.7/41.4
0.5
0.7
ViT-Small [ 2 ]
27.3/28.6
27.8/30.1
28.0/30.9
28.1/31.9
29.9/33.8
31.4/35.1
28.7/31.7
58.7/42.5
22.6
11.2
WReN [ 40 ]
26.8/30.0
27.6/32.0
28.5/32.9
30.1/34.1
36.4/36.3
42.3/39.0
32.0/34.1
64.5/52.5
1.5
0.3
SCL [ 29 ]
25.8/26.9
25.8/30.0
28.3/30.3
34.1/30.0
43.2/31.4
46.2/33.4
33.9/30.3
56.9/53.2
0.2
0.1
IN-ViT-Small [ 2 ]
27.9/31.7
28.2/33.6
28.6/35.3
30.0/37.8
35.6/40.3
47.2/43.6
32.9/37.0
72.4/60.5
22.6
11.2
TABLE II: State-of-the-art comparison on CVR [ 34 ] under joint ( left ) and individual ( right ) training settings. PR-A 2 CL outperforms all baselines across all sample sizes with acceptable cost.
Fig. 4: Comparison of reasoning accuracy on specific compositional rules. PR-A 2 CL outperforms DBCR [ 35 ] for almost all rules.
Human
PR-A 2 CL, 20
PR-A 2 CL, 1k
Elementary
97.6
71.8
99.3
Compositional
74.0
39.2
89.1
Color & Position
57.5
26.2
98.3
Size & Contact
60.0
61.5
95.4
Contact & Rotation
30.0
65.1
94.0
Color & Flip
45.0
27.4
88.5
TABLE III: Human vs. PR-A 2 CL on CVR [ 34 ] with 20/1k samples per task, evaluated on 9 elementary and 36 compositional rules.
Model
20
50
100
200
500
1,000
10,000
ViT-small [ 2 ]
10.1
10.1
10.4
10.5
11.1
11.7
14.1
SCL [ 29 ]
10.1
10.0
10.1
10.2
10.5
12.5
30.4
MRNet [ 58 ]
10.5
10.7
10.9
11.2
11.7
13.5
31.6
WReN [ 40 ]
10.6
10.5
11.0
11.0
11.6
12.6
37.6
SSL-ViT-small [ 47 ]
15.9
16.6
17.1
17.6
18.7
21.7
38.9
SCAR [ 41 ]
10.2
10.1
10.2
10.8
11.5
15.9
39.0
TABLE IV: State-of-the-art comparison on MC 2 R [ 37 ] across varying training sample sizes.
TABLE V: Ablation study of major components on CVR [ 34 ] .
200
500
1000
No CL
66.5
77.8
85.4
WDA Only
70.8
82.2
89.1
SDA Only
67.9
79.4
87.2
A 2 CL
73.2
84.5
91.8
TABLE VI: Ablation of data augmentation strategies.
K
1
2
3
4
Accuracy (%)
91.2
91.4
91.8
91.1
λ
0.02
0.05
0.10
0.20
Accuracy (%)
91.5
91.4
91.8
91.4
TABLE VII: Ablation study of key parameters K and λ .
Fig. 5: t-SNE visualization of feature representations across successive PARBs, showing increased cluster compactness with deeper PARBs.
Fig. 6: Two failure cases on the CVR dataset: 1) rotation & count : DBCR [ 35 ] fails to capture the specific compositional rule. In contrast, the proposed PR-A 2 CL successfully identifies the outlier. 2) position & flip : Both DBCR [ 35 ] and PR-A 2 CL fail to infer the underlying rule, primarily due to interference from random noise introduced by the rotation operation.
School of Artificial Intelligence, Shenzhen University, China · School of Computer Science, University of Nottingham Ningbo China, China · Nottingham Ningbo China Beacons of Excellence Research and Innovation Institute, China