Validation-Frontier Representation Selection under Constrained Observation

Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Search-based methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research.

Learned representations are commonly evaluated through predictive performance, calibration, robustness, uncertainty estimation, and behavior under distribution shift. Yet these criteria do not determine whether the representation itself remains sufficient for organizing observations relevant to an explanatory task. We develop VER (Vigilance Explicative et Representationnelle; explanatory and representational vigilance) around a narrower question: can a system detect when its current representation becomes explanatorily insufficient without confusing insufficiency with ordinary error, missing data, uncertainty, or distribution shift? The current VER framework follows seven stages: TRACE, EVIDENCE, INSUFFICIENCY, REGIME HYPOTHESES, TRANSITION ASSESSMENT, VIGILANCE, and PROBE. It treats structured residuals as signals rather than verdicts, distinguishes data insufficiency from representation insufficiency, preserves multiple hypotheses under non-identifiability, allows a legitimate NON-DETERMINED outcome, and requires a discriminating Probe before stronger conclusions are drawn. It also separates variation, drift, adaptation, and representational transition, and introduces Present Enrichment before attributing explanatory value to history. Two paired thought experiments and a falsifiable benchmark are proposed. VER is not claimed to be empirically validated here; its contribution is an explicit architecture designed for auditability and for selecting observations that discriminate among competing explanations.

Benchmarks of machine learning models often include many datasets, making evaluation expensive. For efficiency, it is preferable to perform evaluations on small, representative datasets instead. The selection of such subsets typically relies on heuristics and is rarely analyzed for the robustness of the resulting model rankings. We introduce a framework to perform the task of selecting datasets subsets with an evaluation of how different selection strategies preserve the global model rankings. Our framework includes bootstrap aggregation, which provides valid confidence intervals, allowing a principled comparison of selection strategies. We consider clustering, design criteria (A/D-optimality), random baselines, and greedy farthest-first (FAFI). For the latter, we derive upper bounds on selection quality in terms of ranking errors as a function of the number of selected datasets. Empirically, in time series classification (TSC, 112 datasets) and in a supplementary natural language processing benchmark derived from MTEB (57 tasks), several selection strategies improve rank preservation compared with random subsets, including simple FAFI. In contrast, in recommender systems (30 datasets), the improvement of strategies over random selection is small and typically statistically insignificant. For TSC, our best-performing strategy achieves a Spearman correlation of 0.95 with the full benchmark model rankings using only five selected datasets. Additional experiments indicate that the effectiveness of selection approaches depends on both the quality of dataset representations and the scale of the benchmarking regime.