VIBES -- Vision Backbone Efficient Selection [2410.08592]