Guided Group Featurizers (GGF): Improving Block-Sparse Featurizers with Object-Guided Learning
Organizations: Algoverse AI Research
Abstract
Block-sparse featurizers (BSFs) use groups of learned directions to capture the multidimensional structure of visual concepts. However, grouping alone does not ensure that information about a concept is consolidated within a single group. We introduce Guided Group Featurizers (GGF), a novel object-guided extension that uses object annotations for initialization and training while preserving the BSF architecture. We assess whether each group's strongest response regions share an object type, part or material across images. Blinded reviews give GGF groups an average semantic coherence score of %, versus % for SLiCS and % for supervised Vanilla BSF. GGF's consolidation advantage extends to unguided categories: one selected group per category covers % of target-containing crops, versus % for supervised Vanilla BSF, at approximately 5% false-positive rates for both. Across COCO and VOC, GGF also improves single-group object Capture and localization over Vanilla BSFs, SLiCS and Concept-SAE. Its groups produce more focused and reproducible object responses while supporting recognition and retaining associations under rotation, scale and brightness changes. GGF also uses fewer learned parameters than Concept-SAE and shorter recorded VOC training loops than both supervised alternatives. These findings support object guidance for organizing more interpretable BSF groups and show that response concentration extends to categories excluded from guidance.
Figures & tables
| Method | Meaning | Visual | Whole |
|---|---|---|---|
| All category labels | |||
| GGF | |||
| SLiCS | |||
| Supervised Vanilla | |||
| Vanilla BSF | |||
| Category-split guidance | |||
| Arm | Capture | Patch AP | VE |
|---|---|---|---|
| Vanilla | |||
| Mask + rec. | |||
| Fixed guidance | |||
| GGF |
Appendix figures & tables59 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Capture | Patch AP | VE |
|---|---|---|---|
| Vanilla | |||
| Supervised | |||
| GGF |
| Seed | Method | Capture | Patch AP | VE |
|---|---|---|---|---|
| 21 | GGF | 40.93 | 77.06 | 81.61 |
| 21 | SLiCS | 26.25 | 52.26 | 75.33 |
| 22 | GGF | 40.85 | 77.39 | 82.52 |
| 22 | SLiCS | 24.51 | 49.69 | 75.49 |
| 23 | GGF | 40.93 | 77.19 | 82.10 |
| 23 | SLiCS | 25.26 | 53.56 | 75.40 |
| Seed | Method | Capture | Patch AP | VE |
|---|---|---|---|---|
| 21 | GGF | 44.21 | 88.09 | 57.08 |
| 21 | SLiCS | 22.41 | 77.18 | 34.27 |
| 22 | GGF | 44.23 | 87.92 | 57.04 |
| 22 | SLiCS | 23.94 | 67.15 | 35.04 |
| 23 | GGF | 44.37 | 88.12 | 57.18 |
| 23 | SLiCS | 22.54 | 70.82 | 35.02 |
| Metric | Gain SD | 95% interval |
|---|---|---|
| Capture | [19.12, 23.50] | |
| Patch AP | [3.91, 28.74] | |
| VE | [21.26, 23.38] |
| Method | Capture | Patch AP | VE |
|---|---|---|---|
| Vanilla BSF | |||
| Supervised Vanilla | |||
| GGF | |||
| Concept-SAE |
| Seed | Model | Capture | AP | VE |
|---|---|---|---|---|
| 21 | GGF | 42.48 | 80.46 | 77.29 |
| 21 | Vanilla | 20.82 | 51.03 | 80.66 |
| 22 | GGF | 42.28 | 80.54 | 76.69 |
| 22 | Vanilla | 20.48 | 52.71 | 79.82 |
| 23 | GGF | 42.24 | 80.33 | 76.49 |
| 23 | Vanilla | 21.90 | 55.24 | 79.77 |
| Seed | Model | Capture | AP | VE |
|---|---|---|---|---|
| 21 | GGF | 42.88 | 74.60 | 77.24 |
| 21 | Vanilla | 21.05 | 49.97 | 80.70 |
| 22 | GGF | 42.76 | 74.87 | 76.82 |
| 22 | Vanilla | 20.73 | 52.00 | 79.94 |
| 23 | GGF | 42.67 | 74.80 | 76.75 |
| 23 | Vanilla | 22.31 | 53.01 | 80.41 |
| Seed | Capture [interval] | AP [interval] | VE |
|---|---|---|---|
| 21 | 21.82 [21.29,22.34] | 24.64 [23.09,26.23] | |
| 22 | 22.03 [21.47,22.51] | 22.86 [21.40,24.52] | |
| 23 | 20.35 [19.83,20.84] | 21.79 [20.31,23.34] |
| Model | Core (M) | Heads | Active |
|---|---|---|---|
| Vanilla | 4.722 | 0 | 16.00 |
| Grassmannian | 2.360 | 0 | 16.00 |
| Group Lasso | 4.723 | 0 | |
| Mask + rec. | 4.722 | 0 | 16.00 |
| Mask GGF | 4.722 | 150 | 16.00 |
| Seed | Model | Capture | AP | VE |
|---|---|---|---|---|
| 21 | Vanilla | 21.178 | 58.722 | 65.196 |
| 21 | Grassmannian | 16.437 | 60.261 | 64.973 |
| 21 | Group Lasso | 20.764 | 56.757 | 64.564 |
| 21 | Mask + rec. | 26.483 | 68.025 | 64.934 |
| 21 | Mask GGF | 32.729 | 71.552 | 64.789 |
| 22 | Vanilla | 20.393 | 60.640 | 65.293 |
| Seed | Stage | Capture | 95% interval | AP |
|---|---|---|---|---|
| 21 | Init. | 5.31 | [4.90, 5.74] | 9.30 |
| 21 | Guided | 6.25 | [6.08, 6.40] | 3.53 |
| 22 | Init. | 6.13 | [5.75, 6.52] | 9.06 |
| 22 | Guided | 6.57 | [6.40, 6.74] | 2.78 |
| 23 | Init. | 5.58 | [5.20, 5.91] | 10.26 |
| 23 | Guided | 6.28 | [6.12, 6.43] | 2.44 |
| Seed | Arm | Capture | AP | VE |
|---|---|---|---|---|
| 21 | V | 21.05 | 49.97 | 80.70 |
| 21 | I | 23.61 | 68.73 | 79.80 |
| 21 | F | 42.38 | 74.41 | 77.54 |
| 21 | G | 42.88 | 74.60 | 77.24 |
| 22 | V | 20.73 | 52.00 | 79.94 |
| 22 | I | 23.53 | 68.38 | 79.33 |
| Change | Capture | Patch AP | Reconstruction VE |
|---|---|---|---|
| Add initialization | |||
| 95% interval | |||
| Add fixed guidance | |||
| 95% interval | |||
| Use adaptive weights | |||
| 95% interval |
| Arm | Capture | AP | VE | Passes |
|---|---|---|---|---|
| GGF 3 fixed | 0/3 | |||
| GGF 4 fixed | 3/3 | |||
| GGF 3 adaptive | 0/3 | |||
| GGF 4 adaptive | 3/3 | |||
| Vanilla BSF 4 | 3/3 |
| Seed | Arm | Capture | AP | VE |
|---|---|---|---|---|
| 21 | GGF 3 fixed | 32.729 | 71.552 | 64.789 |
| 21 | GGF 4 fixed | 33.786 | 72.263 | 66.317 |
| 21 | GGF 3 adaptive | 36.804 | 74.278 | 64.251 |
| 21 | GGF 4 adaptive | 37.727 | 74.879 | 65.740 |
| 21 | Vanilla BSF 4 | 21.837 | 57.932 | 66.761 |
| 22 | GGF 3 fixed | 33.095 | 72.473 | 64.852 |
| Model | Capture | Patch AP | VE |
|---|---|---|---|
| GGF 4: cap 0.40 | |||
| GGF 4: cap 0.20 | |||
| Vanilla BSF 4 | |||
| Original GGF 3 |
| Seed | Capture | Patch AP | VE |
|---|---|---|---|
| 21 | 38.823 | 75.217 | 65.331 |
| 22 | 38.876 | 74.927 | 65.388 |
| 23 | 38.796 | 75.423 | 65.423 |
| Configuration | Model | Updates | Capture | Patch AP | VE |
|---|---|---|---|---|---|
| 5D starting | GGF | 13,640 | |||
| 5D starting | Vanilla | 13,640 | |||
| 5D continued | GGF | 16,640 | |||
| 5D continued | Vanilla | 16,640 | |||
| 6D | GGF | 16,640 | |||
| 6D | Vanilla | 16,640 |
| Metric | ||
|---|---|---|
| Object contrast Capture | 21.32 | 20.43 |
| Patch detection AP | 61.99 | 62.18 |
| Reconstruction VE | 65.21 | 65.43 |
| Core parameters (M) | 4.72 | 18.89 |
| Width | Active groups | Groups | Directions | Active coordinates | Selected update | Capture (%) | Patch AP (%) | VE (%) |
|---|---|---|---|---|---|---|---|---|
| 3 | 16 | 1024 | 3072 | 48 | 2500 | 30.08 | 63.34 | 64.66 |
| 2 | 8 | 1024 | 2048 | 16 | 2500 | 28.66 | 60.84 | 56.46 |
| 2 | 16 | 1024 | 2048 | 32 | 2500 | 27.45 | 60.84 | 62.88 |
| 2 | 32 | 1024 | 2048 | 64 | 500 | 21.14 | 59.29 | 69.65 |
| 3 | 8 | 1024 | 3072 | 24 | 2500 | 30.52 | 59.15 | 57.92 |
| 3 | 32 | 1024 | 3072 | 96 | 0 | 15.08 | 59.46 | 72.65 |
| Model | Locked | Test-matched | VE |
|---|---|---|---|
| Vanilla | |||
| Grassmannian | |||
| Group Lasso | |||
| GGF selected | |||
| GGF final |
| Model or metric | Seed 0 | Seed 1 | Seed 2 |
|---|---|---|---|
| Vanilla recovery | 96.20 | 96.89 | 97.25 |
| Grassmannian recovery | 86.57 | 87.65 | 86.13 |
| Group Lasso recovery | 94.82 | 91.72 | 93.34 |
| GGF recovery | 98.231 | 98.232 | 98.232 |
| Vanilla VE | 97.368 | 97.417 | 97.332 |
| Grassmannian VE | 92.001 | 92.252 | 91.895 |
| Method | One group | Two groups | Four groups | All groups | Selected removal | Random removal |
|---|---|---|---|---|---|---|
| Vanilla BSF | ||||||
| Supervised Vanilla | ||||||
| GGF |
| Coefficient | Contribution | Off-target | |
|---|---|---|---|
| Method | entropy rank | entropy rank | change |
| Vanilla BSF | |||
| Supervised Vanilla | |||
| GGF |
| Category | Vanilla | Supervised Vanilla | SLiCS | Concept-SAE | GGF |
|---|---|---|---|---|---|
| Person | |||||
| Car | |||||
| Dog | |||||
| Chair | |||||
| Potted plant | |||||
| Vase |
| Category | Transformation | Vanilla | Supervised Vanilla | GGF |
|---|---|---|---|---|
| Person | Rotation | |||
| Person | Scale | |||
| Person | Brightness | |||
| Car | Rotation | |||
| Car | Scale | |||
| Car | Brightness |
| Method | Capture (%) | Patch AP (%) | Response reproducibility | Capture reproducibility | Category share (%) |
|---|---|---|---|---|---|
| GGF | |||||
| Supervised Vanilla | |||||
| Vanilla BSF | |||||
| SLiCS | |||||
| Concept-SAE |
| Method | Contribution | Response |
|---|---|---|
| GGF | ||
| Supervised Vanilla | ||
| Vanilla BSF | ||
| SLiCS |
| Specific-pair meaning | |||||||
|---|---|---|---|---|---|---|---|
| Method | 21 | 22 | 23 | 24 | 25 | 26 | 27 |
| Category-split guidance | |||||||
| GGF | 81.78 | 78.00 | 79.33 | 81.33 | 77.56 | 79.56 | 81.56 |
| Sup. Vanilla | 68.89 | 77.33 | 72.67 | 70.67 | 70.00 | 75.11 | 74.67 |
| Vanilla BSF | 54.67 | 63.11 | 61.33 | 58.67 | 58.22 | 68.44 | 58.67 |
| All category labels | |||||||
| Theme outcome | ||||||||
|---|---|---|---|---|---|---|---|---|
| Setting | Method | Clear | Partial | No theme | Insufficient | Unresolved | Disagreements | Unknown votes |
| Split | GGF | 869 | 177 | 4 | 0 | 0 | 37 | 0 |
| Split | Supervised Vanilla | 792 | 245 | 11 | 0 | 2 | 75 | 1 |
| Split | Vanilla BSF | 338 | 176 | 11 | 0 | 0 | 57 | 0 |
| All | GGF | 522 | 3 | 0 | 0 | 0 | 8 | 0 |
| All | SLiCS | 510 | 15 | 0 | 0 | 0 | 25 | 0 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| SLiCS | 1 | — | — | |||
| SLiCS | 2 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| Supervised Vanilla | 1 | — | — | |||
| Supervised Vanilla | 2 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| Supervised Vanilla | 1 | — | — | |||
| Supervised Vanilla | 2 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| SLiCS | 1 | — | — | |||
| SLiCS | 2 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| Supervised Vanilla | 1 | — | — | |||
| Supervised Vanilla | 2 |
| Method | Groups | Coverage | False pos. | Precision | Pos. overlap | Neg. overlap |
|---|---|---|---|---|---|---|
| GGF | 1 | — | — | |||
| GGF | 2 | |||||
| GGF | 4 | |||||
| GGF | 8 | |||||
| Supervised Vanilla | 1 | — | — | |||
| Supervised Vanilla | 2 |
| Full-grid Patch AP | ||||||||
|---|---|---|---|---|---|---|---|---|
| Method | Mean SD | 21 | 22 | 23 | 24 | 25 | 26 | 27 |
| All category labels | ||||||||
| GGF | 73.75 | 74.45 | 74.60 | 74.43 | 74.31 | 74.43 | 74.70 | |
| SLiCS | 49.43 | 49.14 | 51.67 | 50.47 | 48.61 | 50.40 | 49.81 | |
| Supervised Vanilla | 33.25 | 28.70 | 31.97 | 30.83 | 34.86 | 29.93 | 28.30 | |
| Vanilla BSF | 19.61 | 20.37 | 21.47 | 23.16 | 21.08 | 20.64 | 20.38 | |
| Setting | Method | MSE | VE |
|---|---|---|---|
| All labels | GGF | ||
| All labels | SLiCS | N/A | N/A |
| All labels | Supervised Vanilla | ||
| All labels | Vanilla BSF | ||
| Category split | GGF | ||
| Category split | Supervised Vanilla |
| Method | Specific pairs | Visual pairs | Whole clear | Whole + discr. | Rejections |
| All category labels | |||||
| GGF | 0/1 | ||||
| SLiCS | 2/8 | ||||
| Supervised Vanilla | 0/18 | ||||
| Vanilla BSF | 3/27 | ||||
| Category-split guidance | |||||
| Method | Params (M) | Update min | GPU peaks |
| T4: 1,024 groups for BSF arms | |||
| Vanilla BSF | 18.89 | 0.74/0.85 | |
| Supervised Vanilla | 18.89 | 0.87/2.10 | |
| GGF | 18.89 | 1.04/2.21 | |
| Concept-SAE | 92.00 | 1.32/1.48 | |
| A100: 19-group SLiCS comparison | |||
| Method | Parameters | Update min |
|---|---|---|
| GGF | 1,383,452 | |
| SLiCS | 691,200 |
| Method | Update loops | Fit invocation |
|---|---|---|
| GGF | ||
| SLiCS |
| Phase | Images | Call (ms) | Peak GPU (GiB) | GFLOPs per image |
| GGF (19) | ||||
| Inference | 8 | 0.027/0.041 | 0.137 | |
| Ordinary update | 8 | 0.057/0.068 | 0.415 | |
| Controller update | 8 | 0.058/0.070 | 0.663 | |
| SLiCS (19) | ||||
| Category NNLS | 8 | 0.042/0.062 | 0.742 | |
Visual Representation Learning
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
See all 244 papers