Organizations: Waseda University · Waseda Research Institute for Science and Engineering · National Institute of Advanced Industrial Science and Technology · University of Oxford
Formula-Driven Supervised Learning (FDSL) enables vision pre-training without real images, but existing methods cast pre-training as classification on discretized labels, discarding continuity in the generative process. We propose Formula-Driven Supervised Metric Learning (FDSML), which derives relational supervision from image-generation parameters beyond discrete class labels. FDSML combines FineFractalDB, our refined fractal dataset with higher intra-class variation and more stable structures, with a Parameter-Aware Triplet (PA-Triplet) Loss that assigns positives and negatives according to distances in the underlying Iterated Function System parameter space. This objective trains embeddings to respect parameter-derived relative similarity orderings. Experiments show that FDSML improves transferability to image retrieval while remaining competitive on downstream classification and improving robustness under several corruption settings. On ResNet-50, Recall@1 improves from 65.6% to 74.0% on Stanford Online Products and from 56.2% to 78.4% on Stanford Cars over FDSL pre-training. These results suggest that parameter-derived relational supervision provides a stronger synthetic pre-training signal than label-based classification for transfer beyond recognition.
Figures & tables
Figure 1: Overview of the proposed framework with the Parameter-Aware Triplet (PA-Triplet) Loss. (1) An anchor image xa and two candidate images x1 and x2 are sampled from the continuous IFS parameter space together with their associated parameter descriptors Ψa , Ψ1 , and Ψ2 . (2) The candidates are assigned as positive or negative according to their parameter-space distances to the anchor, where dΨ(i,j)=∥Ψi−Ψj∥2 . (3) The resulting triplet (xa,xp,xn) is encoded by a feature extractor into embeddings (za,zp,zn) , and the network is optimized so that the embedding-space distances reflect the relative ordering induced by the underlying IFS parameter space.
Figure 2: Conceptual illustration of the coarse filtering strategy used to construct candidate triplets. (A) all : Two candidates are randomly drawn from the entire dataset. (B) each : One candidate is drawn from the same discrete class as the anchor, and the other from a different class. (C) same : Both candidates are drawn from the same discrete class as the anchor. (D) diff : Both candidates are drawn from classes other than the anchor’s class. Discrete labels only restrict the candidate pool; the final positive and negative assignments are determined by parameter-space distance orderings.
Figure 3: Comparison of FractalDB [ 29 ] and FineFractalDB. FractalDB-1k generates 1,000 instances per class from 25 parameter perturbations with flips and rendering variations. The blue box (left) shows samples differing only by flipping (limited intra-class diversity), while the red box (right) shows degenerate samples from non-contractive transformations. FineFractalDB instead uses independent perturbations with strict contraction constraints, producing richer structural variation and far fewer collapsed patterns.
Filtering
Agreement [%]
dΨ
Label only
all
47.2
-
each
89.2
89.4
same
80.6
50.0 †
diff
46.8
50.0 †
Table 1: Agreement [%] between the dΨ and Chamfer distance orderings under each filtering strategy. Label only assigns the positive by class label ( † : at chance by definition). The value for all is omitted (–), as it would reflect the class composition of the dataset rather than information extractable from the labels.
PT Dataset
PT Type
C10
C100
IN100
IN1k
P30
Scratch
-
87.60
62.70
72.42
75.47
73.50
ImageNet-1k
SL
96.96
84.48
-
-
80.33
FractalDB-1k
FDSL
90.50
70.21
79.42
70.43
76.33
FineFractalDB
FDSML (all)
90.79
68.49
79.62
71.37
76.07
FineFractalDB
FDSML (each)
90.53
67.46
78.80
71.21
77.53
FineFractalDB
FDSML (same)
91.47
68.74
80.62
72.65
77.20
Table 2: Top-1 classification accuracy (%) on ResNet-50.
PT Dataset
PT Type
C10
C100
IN100
IN1k
P30
Cars
Flowers
Scratch
-
76.47
58.02
70.08
71.46
81.17
21.30
48.74
ImageNet-1k
SL
97.92
85.65
-
-
81.48
91.26
93.70
FractalDB-1k
FDSL
96.65
81.18
84.40
72.40
80.46
77.01
73.99
FineFractalDB
FDSML (all)
97.71
81.73
88.47
73.68
80.25
80.83
66.99
FineFractalDB
FDSML (each)
98.00
82.08
88.90
73.71
80.35
83.17
74.21
FineFractalDB
FDSML (same)
97.71
81.68
88.28
73.80
80.58
81.72
69.23
Table 3: Top-1 classification accuracy (%) on ViT-Tiny.
PT Dataset
PT Type
C10
C100
IN100
Scratch
–
90.03
66.83
87.64
ImageNet-1k
SL
98.49
85.73
–
FractalDB-1k
FDSL
98.19
83.68
89.59
FineFractalDB
FDSML (all)
97.96
81.79
89.28
FineFractalDB
FDSML (each)
98.23
83.30
90.09
FineFractalDB
FDSML (same)
98.12
82.19
89.52
Table 4: Top-1 classification accuracy (%) on ViT-Base.
PT Type
PT Dataset
C10
C100
IN100
FDSL
FractalDB-1k
96.65
81.18
84.40
FDSL
FineFractalDB
96.11
77.52
84.32
FDSML (each)
FractalDB-1k
97.77
81.72
87.93
FDSML (each)
FineFractalDB
98.00
82.08
88.90
FDSML (same)
FractalDB-1k
97.16
79.76
87.23
FDSML (same)
FineFractalDB
97.71
81.68
88.28
Table 5: Ablation study on pre-training configurations. Top-1 accuracy (%) of ViT-Tiny on downstream datasets.
Table 6: Classification Error (CE ↓ , %) on ImageNet-C. Each entry reports the average CE over corruption types within each group.
PT Dataset
PT Type
SOP
Stanford Cars
In-Shop
1
10
100
1
4
16
1
10
20
Scratch
-
70.4
83.1
90.9
40.9
61.5
80.4
87.2
95.7
96.9
ImageNet-1k
SL
79.1
91.0
96.3
87.7
95.6
98.6
91.1
97.8
98.4
FractalDB-1k
FDSL
65.6
81.3
91.0
56.2
76.3
89.8
87.3
95.5
96.6
FineFractalDB
FDSML (all)
73.9
86.3
93.3
77.8
89.5
95.4
89.2
96.9
97.9
FineFractalDB
FDSML (each)
74.0
86.5
93.4
76.2
88.6
95.2
89.2
96.7
97.8
Table 7: Recall@K [%] on Stanford Online Products (SOP) [ 60 ] , Stanford Cars [ 33 ] , and In-Shop Clothes [ 39 ] using a ResNet-50 backbone under different pre-training strategies.
Figure 4: Grad-CAM visualizations for ResNet-50 pre-trained under different methods.
Figure 5: Layer-wise CLS-to-patch attention maps of ViT-Tiny pre-trained with different objectives. Rows correspond to different pre-training strategies and columns correspond to Transformer layers ordered from shallow (left) to deep (right).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Response of the Chamfer distance D to controlled perturbations of a single base IFS parameter set. One affine coefficient is rescaled across all maps by a factor s∈[0.6,1.4] (33 steps), and the selection probabilities are recomputed from the determinant magnitudes. Each panel corresponds to one of the six coefficients. The dashed line indicates the rendering noise floor, estimated by re-rendering the unperturbed parameters with different random seeds. D increases monotonically on both sides of the base in all six directions.
Figure B : Chamfer distance D versus descriptor distance dΨ for the perturbation sweeps of Fig. A , colored by the perturbed coefficient. Each sweep contributes two branches corresponding to shrinking ( s<1 ) and enlarging ( s>1 ) perturbations. The rank correlation is nearly perfect within each direction, whereas the slope differs across directions, so dΨ orders structural change reliably but is not calibrated in magnitude across directions.
Figure C : Triplets sampled from FineFractalDB under the four filtering strategies (Sec. 3.4 of the main paper). Each cell shows an anchor and two candidates, with the positive and negative assigned by the descriptor distance dΨ . same yields structurally close within-class candidates, each contrasts a within-class candidate with a cross-class one, and all and diff mostly produce candidates structurally unrelated to the anchor. Fine details are best observed when zoomed in.
Figure D : Randomly selected samples from FractalDB. Each row corresponds to a class and each column shows different instances belonging to that class. Fine details are best observed when zoomed in.
Figure E : Randomly selected samples from the proposed FineFractalDB. Each row corresponds to a class and each column shows different instances generated from independently sampled parameters within the class. Fine details are best observed when zoomed in.
Figure F : Grad-CAM visualizations for ResNet-50 models trained under different pre-training strategies. The leftmost panel shows the input images, while the subsequent panels present the corresponding Grad-CAM heatmaps produced by the ImageNet-1k pre-trained model, the FDSL-pre-trained model, and the proposed FDSML-pre-trained model. For the FDSL and FDSML settings, the models are first pre-trained on synthetic fractal datasets and subsequently fine-tuned on ImageNet-1k. Within each panel, rows correspond to object classes and columns correspond to different instances from the same class.
Figure G : Histogram of cosine similarity between Grad-CAM maps produced by models pre-trained with different strategies and those from an ImageNet-pre-trained ResNet-50, used here as a reference model. Each similarity is computed per image using cosine similarity between Grad-CAM maps. Dashed vertical lines indicate the mean similarity for each method. FDSML variants exhibit higher similarity to the reference model than FDSL, suggesting that their activation patterns tend to align more closely with those of the natural-image pre-trained model.
Figure H : Layer-wise CLS-to-patch attention maps of Vision Transformer models after pre-training under different strategies. The leftmost column shows the input images, and the subsequent columns visualize attention maps from successive Transformer layers. Fine details are best observed when zoomed in.
Figure I : UMAP visualization of feature embeddings from ResNet-50 models pre-trained with different strategies. Features from the layer before the final fully connected layer are projected into two dimensions using UMAP [ mcinnes2018umap-software ] . Ten classes from the FineFractalDB test set are randomly selected and color-coded. Compared with FDSL, FDSML (each) forms more structured intra-class trajectories while remaining relatively well separated across classes, whereas FDSML (same) exhibits more noticeable overlap between classes. Points (A)–(D) indicate representative samples along a trajectory within the same class, with the corresponding fractal images shown below each panel.
Figure J : Visualization of embedding trajectories induced by linear interpolation in the generative parameter space. Two parameter sets are randomly selected from FineFractalDB, and intermediate parameters are generated by linear interpolation with coefficient α∈[0,1] , sampled at 5000 evenly spaced intervals. Fractal images rendered from these parameters are then encoded using ResNet-50 models pre-trained with FDSL or FDSML (same), and the resulting embeddings are projected to two dimensions using UMAP. Points are colored according to the interpolation coefficient α . While the projected embeddings of FDSL exhibit an abrupt trajectory transition around α≈0.6 , those of FDSML (same) form a smoother trajectory that more closely follows the gradual parameter interpolation.
PT Dataset
PT Type
Noise
Blur
Weather
Digital
Geometric
FractalDB-1k
FDSL
39.69
35.49
27.07
22.15
21.55
FineFractalDB
FDSML (all)
38.24
27.43
22.73
19.37
18.84
FineFractalDB
FDSML (each)
43.24
27.08
22.85
20.03
18.73
FineFractalDB
FDSML (same)
41.15
27.65
22.24
19.69
18.47
FineFractalDB
FDSML (diff)
42.55
28.66
24.06
20.27
19.26
Appendix
Table A : Classification Error (CE ↓ , %) on CIFAR-10-C with ResNet-50. Each entry reports the average CE over corruption types within each group.
PT Dataset
PT Type
Noise
Blur
Weather
Digital
Geometric
FractalDB-1k
FDSL
13.56
13.70
7.82
10.09
9.99
FineFractalDB
FDSML (all)
12.33
9.94
6.24
7.56
7.78
FineFractalDB
FDSML (each)
11.64
10.70
6.33
7.30
8.40
FineFractalDB
FDSML (same)
12.45
10.31
6.15
7.58
7.77
FineFractalDB
FDSML (diff)
13.07
10.14
6.70
7.88
8.03
Appendix
Table B : Classification Error (CE ↓ , %) on CIFAR-10-C with ViT-Tiny. Each entry reports the average CE over corruption types within each group.
Metric learning trains distances or embedding geometries so task-relevant examples are close and irrelevant ones far apart for retrieval, clustering, and recognition.
Deep learning-based 3D medical image segmentation methods relies on large-scale labeled datasets, yet acquiring such data is difficult due to privacy constraints and the high cost of expert annotation. Formula-Driven Supervised Learning (FDSL) offers an appealing alternative by generating training data and labels directly from mathematical formulas. However, existing voxel-based approaches are limited in geometric expressiveness and cannot synthesize realistic textures. We introduce Formula-Driven supervised learning with Implicit Functions (FDIF), a framework that enables scalable pre-training without using any real data and medical expert annotations. FDIF introduces an implicit-function representation based on signed distance functions (SDFs), enabling compact modeling of complex geometries while exploiting the surface representation of SDFs to support controllable synthesis of both geometric and intensity textures. Across three medical image segmentation benchmarks (AMOS, ACDC, and KiTS) and three architectures (SwinUNETR, nnUNet ResEnc-L, and nnUNet Primus-M), FDIF consistently improves over a formula-driven method, and achieves performance comparable to self-supervised approaches pre-trained on large-scale real datasets. We further show that FDIF pre-training also benefits 3D classification tasks, highlighting implicit-function-based formula supervision as a promising paradigm for data-free representation learning. Code is available at https://github.com/yamanoko/FDIF.
Discriminative models are trained with task-specific objectives such as classification, detection, or representation learning, yet their practical utility depends on generalizing beyond the training task to unseen samples from the same domain. This suggests that a discriminative model may retain an implicit trace of the data manifold beyond its task labels. We ask whether this trace can be exposed as a generative signal without task-specific guidance. We introduce the Discriminative Score Function (DSF), which extracts an image-space generative field from the loss-gradient geometry of a pretrained discriminative model. DSF provides a training-free update rule that moves images toward model-consistent visual regions without retraining or task-specific targets. We apply DSF to diverse pretrained discriminative models, including ResNet classifiers, DETR detectors, and DINOv2 ViT encoders. DSF generates unconditional images from noise and supports conditional guidance, editing, inpainting, and explanation through the same field. DSF-generated samples also provide effective zero-shot quantization calibration when real images or labels are unavailable, preserving feature geometry better than data-free calibration baselines. These results show that discriminative training leaves a generative functional signal for synthesis and calibration. Project page: https://junhoo.me/dsf.
Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters. We identify three obstacles at this scale -- projection head dominance, representation bottleneck, and augmentation sensitivity -- and propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision. CA-DSSL combines asymmetric distillation from a frozen DINO ViT-S/16 teacher, multi-scale feature distillation for spatial representations, and a progressive augmentation curriculum. On a MobileNetV2-0.35 backbone (396K parameters) pretrained on CIFAR-100, CA-DSSL reaches 62.7 0.5% linear-probe accuracy (3-seed mean) -- surpassing SimCLR-Tiny by 18 pp, matching SEED (61.7%) with 10 fewer projection parameters (426K vs. 3.15M), and reaching 94.0% of a supervised upper bound. Standard SSL methods (BYOL-Tiny, DINO-Tiny) collapse entirely at this scale. On Pascal VOC detection, CA-DSSL achieves 2.3 the mAP of random initialization and +3 pp over SEED, though SimCLR-Tiny matches CA-DSSL on detection mAP. The deployed backbone occupies 378 KB (INT8) with no inference overhead from pretraining. Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes; scaling to ImageNet-1K is discussed as future work.