When Does Machine Learning Beat Value Sorting? A Three-Dataset Diagnostic of Exposure-Weighted Shipment Prioritization

Routing decisions between a cheap heuristic and an expensive large language model (LLM) are typically framed as a difficulty problem: send the hard cases to the expensive path. We argue this framing is incomplete because difficulty and business value are distinct axes - a difficult cheap item and a difficult costly item do not have the same cost of error. We present Value Router, a fully synthetic simulation of a retail merchandising pipeline that routes items using only estimated difficulty and estimated value, never ground truth. The study has three stages. First, a value-weighted threshold router is compared with a difficulty-only and a random baseline on a synthetic catalog with an inverse correlation between category volume and value. Value-weighting matches the difficulty-only baseline's recall of true high-value items (60%) while achieving substantially higher precision (98.3% vs. 94.3%). Second, a decision logger and monitor expose a failure mode hidden by aggregate metrics showing that the aggregate result is driven almost entirely by between-category differences rather than per-item discrimination. Third, a simulated Black Friday demand surge (2.5 volume with a shift toward higher-value categories) compares a static router, a seasonally tuned router, and two slow-path budget policies. All results are from a controlled synthetic simulation with experimenter-defined ground truth and illustrate design principles for cost-aware routing systems rather than validated real-world claims.

Shapley value and its priority-aware extensions are widely used for valuation in machine learning, but existing methods require pairwise priority to be binary and acyclic, a restriction spectacularly violated in real-data examples such as aggregated human preferences and multi-criterion comparisons. We introduce the generalized priority-aware Shapley value (GPASV), a random order value defined on arbitrary directed weighted priority graphs, in which pairwise edges penalize rather than forbid order violations. GPASV covers a range of classical models as boundary cases. We establish GPASV through an axiomatic characterization, develop the associated computational methods, and introduce a priority sweeping diagnostic extending PASV's. We apply GPASV to LLM ensemble valuation on the cyclic Chatbot Arena preference graph, illustrating that priority-aware valuation is not a one-button operation: different balances of pairwise graph priority versus individual soft priority produce substantively different valuations of the same data.

Organizations often have an incumbent predictive model in production when new data sources become available. Because historical training data lack the new features, a challenger model must be trained on a small but growing full-feature dataset. We study whether, and when, the organization should switch to the challenger. The decision is statistical and economic: the challenger's predictive performance improves as full-feature data accumulate, but repeated retraining is costly and delays benefits from deployment. We develop a framework linking learning-curve dynamics to model-switching economics. Under a standard power-law learning curve and finite data-collection horizon $T$, the optimal time to train and evaluate the challenger scales as $T^{1/(1+α)}$: learning-curve shape (through its learning speed $α$) is the primary theoretical determinant of when to stop experimenting; costs determine switching profitability. Even without knowing the learning curve, the operational problem is tractable: we show that any algorithm stopping on the $T^{2/3}$ scale and making reliable switch/discard decisions achieves $O(T^{2/3}\sqrt{\log T})$ regret relative to a full-foresight oracle. We propose a sequential evaluation algorithm that uses local learning-curve trends to anticipate improvement, and test it in a real-world credit-scoring study. Even with this local approximation, the algorithm theoretically and empirically achieves near-oracle performance. It is also more stable than greedy sequential evaluation algorithms, where noisy early estimates trigger premature discarding, or simple one-shot evaluation algorithms, which work only when their fixed evaluation time matches the (unknown in practice) theoretical timing scale. Our framework offers a step toward principled model governance when new data sources require costly collection, validation, and deployment.