Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not regress any metric. In practice, feature deprecation is hard to prioritize as a feature can fail to be removed if it interacts with other features, its importance changes daily, it cannot be evaluated before production models change or data drift occurs, it is in contention for A/B test space with other launches, it causes regressions in an online A/B test, or it is consumed by multiple models that need to coordinate its deprecation. We present FORTE, a model-agnostic production framework for discovering, validating, and removing features. Its efficient perturbation implementation allows daily feature evaluation. To account for interactions, it evaluates candidate feature sets constructed from all the low-rank features through grouped ablation and its offline results are sufficiently predictive of online A/B tests to allow deprecations to proceed to online testing with confidence. Lineage tracing identifies all downstream consumer models to realize the full savings of removing a feature from the training set. We validate FORTE over 2 large-scale recommender systems and find that it identifies deprecable feature sets which do not cause online regressions, while the impact on efficiency depends on model characteristics. On a search-relevance model, FORTE resulted in 3 nested candidate sets of 32, 48, and 59 features. Online guardrails rejected the largest treatment. The 48-feature set was selected for deprecation, removing 43.3% of the model's features. On a large ranking model, FORTE identified 8 candidates for removal. Retraining and A/B testing confirmed no regression. Its launch reduced the trainable parameters by 29.9%, improved serving throughput by 19.8-25.5%, and reduced annual training costs by hundreds of thousands of dollars.
Figures & tables
Figure 1. FORTE’s deprecation funnel. Candidate sets narrow through validation and deprecation; fill identifies each stage’s bottleneck. A left-to-right funnel of five narrowing stages: temporal feature importance and candidate construction, grouped perturbation, retraining, online A/B, and lineage and deprecation. Each stage is annotated with what it consumes, and fill colour indicates whether the stage is bottlenecked by amortized inference, training capacity, online traffic, or organizational coordination.
Figure 2. Feature-rank trajectories for two features across scheduled observations. Lower ranks denote more highly ranked features. The observations retain their original order; the horizontal labels show the corresponding abbreviated month names. A two-line chart showing feature rank from January through September. Feature A fluctuates between ranks 111 and 267. Feature B remains mostly between ranks 24 and 65 before moving to rank 120 in September. The vertical axis begins at 0 and increases upward.
Figure 3. Dataset virtualization (ablation): the original dictionary holds typed batches; perturbed holds type-appropriate nulls. Experiments draw each feature column ( fn ) from one of these dictionaries. Two feature-keyed dictionaries (original and perturbed) feed into a third box representing one experiment that ablates feature f3: f1 and f2 are pulled from original, f3 from perturbed.
Figure 4. AUC and PR-AUC feature-ranking correlations across approximately matched row-evaluation budgets, measured against a separate 1.5M×1 reference. Tick labels show dataset size followed by the number of permutations in parentheses. A line plot of AUC and PR-AUC feature-ranking correlations for the Closeup and Clickthrough heads against dataset size. Each x-axis tick also gives the corresponding permutation count. All four correlations generally increase as the dataset grows and the number of permutations decreases.
Figure 5. Runtime grows sublinearly with experiment count for all three evaluation-set sizes. Dashed lines show linear scaling from each series’ single-experiment runtime; the 3M-row set was measured through 300 experiments. A log-log plot of wall-clock time against the number of perturbation experiments per run, for evaluation sets of 30k, 300k, and 3M rows. All three series rise far more slowly than the dashed linear-scaling references anchored at each series' single-experiment runtime.
Figure 6. Closeup AUC effects for 189 feature groups, ranked by Individual-Feature Permutation relative Δ AUC. The dashed curve shows individual permutation; the solid curve and bars show the mean and 95% confidence interval over 20 randomized addition orders, computed as t9 intervals over the ten reverse-pair means. The vertical axis uses a symmetric-logarithmic scale. A line plot with feature-group rank from 1 to 189 on the horizontal axis and relative Closeup AUC change on a symmetric-logarithmic vertical axis. A muted dashed line shows Individual-Feature Permutation effects. A prominent solid red line shows the corresponding order-averaged marginal effects, accompanied by light vertical 95 percent confidence-interval bars, and crosses below zero for some feature groups.
Features removed
Teacher acc. Δ
Human acc. Δ
Outcome
Trainable params. Δ
Serving throughput Δ
32
−0.0010
−0.0012
Superseded
—
—
48
−0.0009
+0.0042
Launched ∗
−2.3%
+2.0%
59
−0.0011
−0.0012
Rejected
—
—
Table 1. Search Relevance: offline results and deployment outcomes for the three nested candidate sets, one treatment arm each. All three cleared every offline gate; only the online guardrails separated them. Efficiency gains are reported for the launched 48-feature configuration.
Features removed
Pairwise Accuracy Δ∗
NDCG@3 Δ∗
Outcome
Trainable params. Δ
Serving throughput Δ
Inference cost Δ
2
+0.0001
−0.0015
Superseded
—
—
—
8
+0.0001
−0.0010
Launched
−29.9%
+19.8 – 25.5%
∼−10%
Table 2. Pin Ranker: Retraining results for the two nested candidate sets and efficiency gains from launching the 8-feature removal. Throughput is measured on a CUDA Graph benchmark.
Large-scale ranking systems depend on thousands of features derived from user behavior across multiple time horizons. Typically requires model retraining -- resulting in long iteration cycles (3--6 months), substantial GPU resource consumption, and limited rollout throughput. We introduce Intelligent Elastic Feature Fading (IEFF), a production infrastructure system that enables retrain-free feature efficiency rollouts by elastically controlling feature coverage and distribution at serving time. IEFF supports incremental feature coverage adjustments while models adapt through recurring training, eliminating dependencies on explicit retraining cycles. The system incorporates strict safety guardrails, reversibility mechanisms, and comprehensive monitoring to ensure stability at scale. Across multiple production use cases, IEFF accelerates efficiency-related rollouts by 5×, eliminates retraining-related GPU overhead, and enables faster capacity recycling. Extensive offline and online experiments demonstrate that gradual feature fading prevents 50--55% of online performance degradation compared to abrupt feature removal, while maintaining stable model behavior. These results establish elastic, system-level feature fading as a practical and scalable approach for managing feature efficiency in modern industrial ranking systems.
Modern industrial recommender systems rely on thousands of heterogeneous features -- ranging from low-dimensional scalars (e.g., statistical value) to high-dimensional embeddings (e.g., user-id embeddings, MLP representations) -- to achieve high-precision predictions. Given the immense computational costs associated with training, efficient feature selection is critical. However, existing methods encounter three primary bottlenecks: (1) they typically assume uniform feature dimensions or require costly mapping to a fixed size; (2) they struggle with extreme sparsity, where the majority of features (e.g., 99%+) remain at default values; and (3) traditional permutation-based approaches are computationally prohibitive in large-scale settings. To address these challenges, we propose LeAP (Learnable Adaptive Permutation), a novel, model-agnostic plug-in module for feature selection. LeAP transforms the inefficient random permutation process into a learnable mechanism, significantly accelerating the evaluation of feature importance. In addition, we introduce an adaptive regularization strategy tailored for heterogeneous dimensions and extreme sparsity, enabling superior feature importance ranking results across asymmetric input spaces. Experiments on four public recommendation datasets demonstrate that LeAP achieves state-of-the-art performance. Furthermore, LeAP has been deployed in a large-scale industrial search ranking model with over a billion daily requests and a 2TB model parameter scale. In this real-world scenario involving 12,000+ total feature dimensions, LeAP successfully identified and removed over 3,600 redundant dimensions without performance degradation, which is 2 to 10 times the ability of compared baseline methods.
Modern industrial recommender systems use a deep ranking model to score N candidates against the same user and context features. Standard implementations broadcast context features early in the forward pass, redundantly computing context-only operations N times per request. We present a rank-aware decomposition applicable to the dominant interaction mechanisms in modern recommender architectures-Factorization Machine (FM) pairwise products, Deep Cross Network (DCNv2) cross layers, self-attention, and fully connected (FC) projection layers-built on a single algebraic principle: any linear or bilinear operation over a rank-partitioned input admits an exact block decomposition that moves context-only computation from once-per-candidate to once-per-request, identity-equivalent to the original model. Closed-form analysis and controlled ablation verify that savings scale quadratically with the number of context features. Applied to a production DLRM-style ranker without any architectural change, the decomposition increases per-pod throughput by 87.5% (a 47% reduction in peak pod count) at identical model predictions. The identity-equivalent decomposition applies only at the first layer of cross networks and self-attention, since each layer mixes ranks in its output. To extend savings across depth, we further introduce rDCN, an architectural variant of DCNv2 that maintains rank discipline across depth and matches DCNv2 accuracy within training noise at 67% fewer total FLOPs, and sketch an analogous architectural variant for self-attention.