On-Device Commercial Intent Retrieval Under Size, Latency, and Privacy Constraints: A 3 MiB Retrieval System with Typed Egress Boundaries

Hyojung Han

cs.IRSep 16, 2026

On-Device Commercial Intent Retrieval Under Size, Latency, and Privacy Constraints: A 3 MiB Retrieval System with Typed Egress Boundaries

Authors: Hyojung Han

Abstract

We study commercial intent inference that runs entirely on the user's device, under three constraints frozen before the work began: the downloaded payload under 3 MiB, Tier-0 inference under 20 ms at p95, and no raw text, content embedding, or stable identifier leaving the device. Under them we build a retrieval path over a 6,020-leaf commercial taxonomy: a static embedding table distilled from a Korean sentence transformer, quantized to 4 bits, no inference runtime. Our main result is where that constraint costs accuracy. On real Korean commerce text labelled by others (22,900 AI-Hub shopping reviews), mid-category top-5 on real product names is 75.0% against an 18.4% permutation baseline, but splits on one observable: a query containing some leaf name as a substring scores 83.5%, one containing none 45.2%. A generic 196.6x larger teacher seemed to localize the gap (+20.1 pp without an anchor, +0.1 with). That null was two effects cancelling: the same teacher fine-tuned on the student's own contrastive pairs reaches 0.8586 and beats the pure-encoder student by +10.6 pp with an anchor and +20.9 pp without. The cost is not uniform, but it is not free anywhere; where the anchor is absent, task adaptation buys the teacher nothing, so what the constrained encoder lacks there is capacity. The expensive regime is detectable on-device from the ranker's own score margin: declining the least confident fifth lifts the rest to 0.8296. A second axis we first reported, a manufacturer model code, does not survive source-category fixed effects (-4.0 pp, p=0.51); the anchor does (+13.0 pp). Payload is 2,942,652 bytes, all three library links measured. Tier-0 p95 is 4.431 and 3.670 ms on two iPhones (A14, A16) and 5.080 ms on a budget Android tablet (Snapdragon 695), all slower than three server CPUs on the same code. Taxonomy supervision is mostly synthetic Korean utterances.

Figures & tables

Constraint	Falsifiable claim	Measured
Payload $\leq 3$ MiB	Downloaded bytes, data $+$ library code, $\leq 3{,}145{,}728$	$2{,}942{,}652$ B ( $93.5\%$ ), worst of three measured links (Android); iOS $2{,}926{,}428$ B
Tier-0 latency $<20$ ms	$p_{95}<20$ ms on the reference handsets	$p_{95}$ $\mathbf{4.431}$ ms on an iPhone 12 Pro ( $22.2\%$ ) and $\mathbf{3.670}$ ms on an iPhone 14 Pro Max ( $18.3\%$ ); $5.080$ ms on a Galaxy Tab A9+ ( $25.4\%$ ); $2.00$ – $3.37$ ms on three server CPUs
Zero raw text	No raw token appears in any byte the client sends	Canaries $7/7$ undetected in a $79{,}968$ -byte payload
Zero content embeddings	The upload schema has no field a vector can occupy	Type-checked; § 7
Zero stable identifiers	The ad request type has no field an identifier can occupy	Type-checked; § 8

Table 1: Constraints versus measured values. Every measured value in this paper, here and throughout, comes from the shipping manifest in artifacts/MANIFEST.json and the gate outputs under gates/ , as published at evidence commit 155b6d94c6ce [ 46 ] . The budget limits the gates enforce are in gates/budgets.json at that commit, so the constraints are checkable and not merely asserted.

System	Where inference runs	Intent granularity	Personal learning	Footprint
Adnostic [ 42 ]	browser extension	interest profile over a fixed category set	none	—
Privad [ 43 ]	client $+$ interposed broker	behavioural categories	none	—
Topics [ 45 ]	browser, platform-owned	a few hundred coarse topics	platform-managed	browser-owned
This work	app SDK, no runtime	$6{,}020$ -leaf taxonomy, shown at mid level	federated combiner (design; coefficients pending)	$2.93$ MB measured

Table 2: Where client-side ad targeting systems stop. “Footprint” is the download the client must carry for the targeting path; a dash means the work does not report one.

On the device (raw data never leaves here)

Tier 0 (always on,

2.14

MB of data, no runtime required)

recent text / search query / app context

\rightarrow

tokenize

\rightarrow

static embedding lookup

\rightarrow

mean pooling

\rightarrow

L2 normalize

\;\Rightarrow\;v_{0}

Tier 1 (only when the device has an OS model; SDK cost

0

on iOS)

OS language-model summary / OS dense image embedding

\rightarrow

into the

v_{0}

space via projection

P

or rectangular projection

W_{f}

Figure 1: The three-tier structure. Tier 1 is an OS model running on the device, not a server.

	Tier 0	Tier 1
Availability	$100\%$ . We ship it	Device-dependent
SDK cost	Data $2.14$ MB + code $50$ KB	iOS $0$ B; Android varies by path
Latency	On the order of milliseconds	Hundreds of ms
What it does	Always-on intent vector	Deep interpretation of ambiguous context
If absent	There is no product	Only quality degrades

Table 3: How the two tiers differ.

Slice	Length	Where
Projection head $P$	$4{,}096$	To the server. Aggregated
Combiner $\theta$	$5$	To the server
Combiner MLP (pilot arm)	$89$	To the server
Per-user combiner $\theta_{u}$ (pilot arm)	$8$	To the server
Personal intent head $w$	$64$	Stays on the device

Table 4: The federated vector layout. Under the base arm, the federated portion is

4{,}096

and the personal portion is

64

; including both pilot A/B arms brings the federated portion to

4{,}198

and the total to

4{,}262

. Source: gates/budgets.json .

Tier	Character	What it carries
Service-use consent	Required
Behavioral-data collection consent	Optional	Fixed 12-byte events. The fact of an interaction, such as “tapped”
Screen-content reading consent	Optional, higher tier	The embedding of the actual content the user read or wrote

Table 5: Consent tiers. A higher tier has the lower tier as a precondition.

Metric	Previous	Shipping	Delta (95% CI)
Leaf top-1 (objective)	$0.2104$	$0.2235$	$+1.30$ pp $[+0.63,\ +1.98]$
Leaf top-5	$0.4715$	$0.4965$	$+2.50$ pp $[+1.70,\ +3.28]$
Product surface (L2 top-5)	$0.7767$	$0.7989$	$+2.22$ pp $[+1.48,\ +2.96]$
Catalog creative $\rightarrow$ leaf	$0.7081$	$0.7404$	$+3.22$ pp

Table 6: Evidence for the current deployment’s promotion ( stage6cov985_e10 ) relative to the previous deployment ( stage5cap2k_e10 ). Held-out leaves,

n=4{,}600

, bootstrap

4{,}000

. Source: experiments/distill-ko/promote_gate_cov985_e10.json .

Arm	emb gzip	KLUE-STS	taxo top-1	YNAT	NSMC
native-64 (previous deployment)	$994{,}734$	$0.6103$	$0.2410$	$0.6340$	$0.7365$
native-128 (trained directly)	$1{,}767{,}620$	$0.6881$	$0.4723$	$0.6640$	$0.7542$
pca-trunc 128 (adopted)	$\mathbf{1{,}778{,}722}$	$0.6965$	$\mathbf{0.5137}$	$0.6542$	$0.7585$
mrl-t128	$1{,}786{,}390$	$0.6996$	$0.5020$	$\mathbf{0.6750}$	$0.7560$

Table 7: Comparison of dimensionality and truncation methods. All measured at the deployment quantization, paired bootstrap

n=4{,}000

. emb gzip is the transmitted bytes of the embedding artifact alone. Source: docs/spec/11-dimension-decision.md .

Configuration	L1 gzip	KLUE-STS	YNAT	NSMC
$31{,}497\times 64$ int8 (v1)	$2{,}276{,}098$	$0.5806$	$0.5132$	$0.7215$
$31{,}497\times 128$ int8	$4{,}117{,}221$	$0.6058$	$0.6170$	$0.7500$
$31{,}497\times 512$ @ 4bit	$4{,}002{,}327$	$0.5754$	$\mathbf{0.6870}$	$\mathbf{0.7750}$

Table 8: Vocabulary fixed at

31{,}497

; comparison of dimensionality and bit-width. The size column holds pre-build projections, not measurements — they are the figures the dimension decision was taken on, recorded here because the next subsection is about how the measurement contradicted them (

4{,}002{,}327

projected against

6{,}115{,}038

measured, experiments/dim-vs-vocab/pack_stage2_results.json ). The quality columns are measured. Source: CONTRACT.md and docs/spec/11-dimension-decision.md .

Query distribution	$n$	L2 top-1	L2 top-5	Leaf top-1	Leaf top-5
Unseen categories, colloquial	$4{,}600$	$0.4252$	$\mathbf{0.7757}$	$0.2196$	$0.4739$
All, colloquial	$35{,}830$	$0.4533$	$\mathbf{0.8037}$	$0.2118$	$0.4846$
Verbatim leaf-name queries	$6{,}020$	$0.9309$	$0.9850$	$0.9259$	$0.9786$
Ad creative $\rightarrow$ category	$6{,}020$	$0.8164$	$0.9576$	$0.8073$	$0.9503$

Table 9: Product-surface accuracy. Measured on the deployment path (dense + lexical overlap).

28

top-level,

256

mid-level,

6{,}020

sub-level categories, measured on the shipping 4-bit artifacts (§ 5.5 ). Source: experiments/distill-ko/product_surface.json .

	Leaf top-1	Leaf top-5	Product surface top-5
Deployed (2.14 MB static)	$\mathbf{0.2369}$	$0.3823$	$\mathbf{0.529}$
Teacher (768d, 421 MB)	$0.1741$	n/a	$0.473$
Difference (leaf top-1)	$+6.28$ pp, 95% CI $[+4.57,\ +7.99]$

Table 10: Deployed model versus teacher on real search queries (ESCI Japanese, translated to Korean,

n=1{,}465

held out), measured at the stage5cap2k_e10 build. Over the

2{,}960

-leaf taxonomy that preceded the expansion — leaf ranks are not comparable across taxonomy sizes, so these figures must not be read beside the

6{,}020

-leaf numbers elsewhere in this paper. Source: experiments/esci-ko/esci_eval_results.json .

Axis	8-bit	4-bit	$\Delta$
taxonomy.bin $+$ catalog.bin (gzip)	$1{,}332{,}138$	$906{,}659$	$-425{,}479$ B
Deployment payload (L1 $+$ L2, measured target)	$3{,}352{,}283$	$\mathbf{2{,}926{,}804}$	$-425{,}479$ B
Unseen categories, L2 top-5 (product surface)	$0.7759$	$0.7757$	$-0.02$ pp
Unseen categories, L2 top-1	$0.4304$	$0.4252$	$-0.52$ pp
All colloquial, L2 top-5	$0.8045$	$0.8037$	$-0.08$ pp
Verbatim queries, leaf top-1	$0.9299$	$0.9259$	$-0.40$ pp

Table 11: Dropping category and catalog vectors from 8-bit to 4-bit. Same evaluation, same queries, arms differ only in the artifact quantization. Source: experiments/distill-ko/quant4_surface.json .

Arm	L2 top-5	Leaf top-1
baseline (current deployment)	$0.9510$	$0.4717$
off (new 3-channel weights, no perception input)	$0.9531$	$0.4739$
on (same weights, with perception input)	$0.9548$	$\mathbf{0.4555}$
off $-$ baseline	$+0.22$ pp $[-0.03,+0.48]$	$+0.22$ pp $[-0.30,+0.73]$
on $-$ off	$+0.17$ pp $[-0.13,+0.47]$	$\mathbf{-1.83}$ pp $[-2.45,-1.20]$

Table 12: Categorical perception channel A/B test. Panel of

6{,}000

, holdout of

5{,}995

, channel dropout

0.30

, scored with the deployment function. Source: experiments/user-embedding/tier1_arms.json .

Arm	Accuracy	Upper bound on what
block_prior	$0.3148$	Ceiling of the categorical signal. Even with the block correctly identified $100\%$ of the time, the best one can do inside it is the mode
image_within	$\mathbf{0.8764}$	Dense embedding, candidates restricted to the correct block
image_open	$0.8700$	No candidate restriction
image_master	$0.9882$	Control on the coarse axis

Table 13: Dense-embedding resolution probe. Source: experiments/percept-vision/resolution_probe.json .

	Dense embedding	Cost	Categorical	Cost
iOS	VNGenerateImageFeaturePrintRequest	$0$	Vision classification, text recognition	$0$
Android	MediaPipe ImageEmbedder , model bundled by the app	several MB	ML Kit unbundled	about $200$ KB

Table 14: Perception paths and SDK cost by platform.

	English	Korean	Multi-vertical
Title-arm purity (the ceiling)	$0.3440$	$\mathbf{0.5157}$	$0.3943$
Image-arm purity	$0.1233$	$\mathbf{0.3015}$	$0.2710$
Image $/$ ceiling ratio	$0.358$	$0.585$	$\mathbf{0.687}$
Mid-categories used by the title arm	$102$	$\mathbf{137}$	—
L2 top-5 match	$0.1400$	$\mathbf{0.4214}$	$0.2681$

Table 15: Raising the target raises the projection. Left: the translation axis, single vertical,

n=3{,}600

. Right: re-validation on

456

verticals from Amazon Berkeley Objects [ 37 ] , same procedure. Full arms in the evidence bundle [ 46 ] ( percept-vision-fashion-ko , percept-vision-mv ).

#	Invariant	Why
F1 ∗	The server must not be able to see individual deltas	The privacy accounting in § 7.4 applies only under this condition
F2	Deltas are clipped before upload	Noise without clipping provides no guarantee
F3	The deadline is itself the straggler policy	Waiting for slow devices would prevent the round from ever closing
F4	Late-arriving deltas are discarded	Mixing in deltas computed against a stale $\theta$ causes silent divergence
F5	A round is cancelled if participation falls below the minimum	Aggregating too few participants exposes individuals
F6	Rounds are logged reproducibly	Without this, regressions cannot be caught

Table 16: Federated protocol invariants.

Signal	What it says	Combiner $\theta$	Personal head $w$
Creative tap	The user pointed to one leaf	Positive	Positive
Positive feedback	The whole answer was acceptable	Unused	Positive
Negative feedback	The whole answer was wrong	Unused	Negative
In-session requery	The prior answer was possibly wrong	Unused	Unused

Table 17: Assignment of on-device training signals. Source docs/spec/17-on-device-labels.md .

Assumption	Campaign $\varepsilon$
No amplification ( $q=1$ )	$16.25$ (hand-computed zCDP $17.67$ )
Population $10{,}000$ , cohort $1{,}000$ ( $q=0.1$ )	$\mathbf{1.54}$
Population $100{,}000$ ( $q=0.01$ )	$0.14$
Composed including the attribution channel	$\mathbf{1.7295}\leq 2.0$

Table 18: Privacy budget. Top: campaign-wide

\varepsilon

(PLD accounting, 50 rounds,

\delta=10^{-10}

, effective noise multiplier

z=3.162

). Bottom: per-round

\varepsilon

versus number of participants. The minimum number of participants satisfying the target of

5.0

204

. Source gates/dp_gate.py , artifacts/ui-facts.json .

	Uniform weighting		Correction applied
Arm	Mid-level (L2) top-5	Sub-category top-1	Mid-level (L2) top-5	Sub-category top-1
B: Shipping build + personal head	$+6.19$ pp $[5.55,6.80]$	$+3.74$ pp	$+5.00$ pp $[4.40,5.62]$	$\mathbf{-2.53}$ pp
C: Personal head + federated	$+7.92$ pp $[7.25,8.64]$	$+3.95$ pp	$+6.94$ pp $[6.27,7.62]$	$\mathbf{-1.75}$ pp
D: Federated only	$+7.09$ pp $[6.44,7.75]$	$+2.62$ pp	$+3.57$ pp $[3.04,4.09]$	$-5.14$ pp

Table 19: User embedding arms. Panel

6{,}000

, holdout

5{,}997

, disjoint by person. The baseline is the shipping build. The left two columns use uniformly interest-weighted personas ( ship_arch.json ); the right two columns use personas regenerated with the 2026-09-01 REES46 correction actually applied ( ship_arch_fix.json , after fixing the generator’s binary-search defect).

	Current	Needed	Shortfall factor
Creative-tap labels for training	$3$	$800$	roughly $250\times$
Creative-tap labels for evaluation	$3$	$1{,}750$	roughly $580\times$

Table 20: Shortage of real labels. Source experiments/user-vector-gap/default_on_verdict.json .

Observation	Consequence
$62.6\%$ of evaluation rows had a single prior purchase	$p_{\text{buy}}$ returns the same constant for all of them
Positive rate $90.4\%$ ( $w=7$ d), $46.5\%$ forced to 1	the window truncation is being measured, not the model
Median inter-purchase gap $0.5$ days	one order split across items counted as a repurchase
$79.8\%$ of pairs dropped for having no repurchase	censored users removed from the evaluation

Table 21: Diagnosis of the first harness. Each row is a property of the evaluation task, not of the model. Source experiments/purchase-lift/diagnose_real.py .

Arm	$w{=}3$ d	$\Delta$	$w{=}7$ d	$\Delta$
mPG (deployed)	$0.7327$	—	$0.7000$	—
Weibull $m{=}0.9$	$0.7385$	$+0.0058$ [ $+0.0032,+0.0082$ ]	$0.7026$	$+0.0026$ [ $+0.0012,+0.0038$ ]
① L2 base rate only	$0.5755$	$-0.1572$	$0.6033$	$-0.0967$
② $+$ frequency $k$	$0.6885$	$-0.0442$	$0.6826$	$-0.0174$
③ $+$ purchase recency	$0.7454$	$+0.0127$ [ $+0.0057,+0.0196$ ]	$0.7249$	$+0.0249$ [ $+0.0162,+0.0333$ ]
④ $+$ Weibull logit	$0.7456$	$+0.0129$	$0.7257$	$+0.0257$

Table 22: Ablation on real logs.

\Delta

is against the deployed formula; brackets are 95% bootstrap CIs. Source experiments/purchase-lift/real-v2-ablation-2026-09-05.json .

Arm	KLUE-YNAT (7-way)	NSMC (binary sentiment)
Stage 1 (distill only)	$0.5240$ $[0.514,0.535]$	$0.7283$ $[0.720,0.737]$
Stage 2 (+ tokenlearn)	$\mathbf{0.6408}$ $[0.631,0.651]$	$\mathbf{0.7362}$ $[0.727,0.745]$
Public Korean static embedding (64d)	$0.3987$ $[0.389,0.409]$	$0.6844$ $[0.676,0.694]$
Teacher, 768d (upper bound)	$0.8188$	$0.8498$
Shipping build (128d, 4-bit; § 10.9 )	$\mathbf{0.6750}$	$\mathbf{0.7548}$
Retention relative to teacher (Stage 2)	$78.3\%$	$86.6\%$

Table 23: Real-data Korean classification. Brackets show the 95% bootstrap confidence interval. Source experiments/real-ko-bench/results.json .

Dataset 71603 — mid-category (L2) top-5, the product surface
Query set	$n$	Accuracy	perm.
Real product names, all	$1{,}174$	$0.7504$ $[0.726,0.775]$	$0.1843$
— leaf-name anchor present	$915$	$\mathbf{0.8350}$	—
— anchor absent	$259$	$0.4517$	—
— anchor absent, model-code form	$137$	$0.2847$	$0.1554$
Real review sentences	$6{,}000$	$0.4552$ $[0.4423,0.4677]$	$0.1241$

Table 24: Deployment path on real Korean commerce text. Brackets are 95% bootstrap confidence intervals over

2{,}000

resamples; perm. is the permutation baseline under the identical mapping, averaged over

400

draws. Sources experiments/real-ko-commerce/aihub_surface.json and aihub102_surface.json .

Evidence	What it can settle	What it cannot
Synthetic held-out utterances	Task-aligned accuracy over our taxonomy, regression detection	Authored by us; not a real query distribution
ESCI queries translated to Korean	Behaviour on real production search queries	Amazon’s user distribution in translation; labels derived, not human
AI-Hub 71603	Real Korean product and review text with externally assigned source labels	Not queries; crosswalk to our taxonomy is ours
AI-Hub 102	Real Korean customer utterances, $307{,}551$ of them	Labels support only the top category, not the product surface
Naver Shopping reviews	The distribution of real Korean commerce text	No labels at all; supports distribution comparison only

Table 25: Evidence scope. No row is strong on every axis; the argument rests on their weaknesses being different from one another.

Cell	$n$	top-5	cell-matched null
Anchor present, no model code	$684$	$0.8772$	$0.2793$ ( $+59.8$ pp, $p<0.0025$ )
Anchor absent, no model code	$122$	$0.6393$	$0.1610$ ( $+47.8$ pp, $p<0.0025$ )
Anchor present, model code	$231$	$0.7100$	$0.2363$ ( $+47.4$ pp, $p<0.0025$ )
Anchor absent, model code	$137$	$\mathbf{0.2847}$	$0.1554$ ( $+12.9$ pp, $p<0.0025$ )
Marginal, anchor present	$915$	$0.8350$	—
Marginal, anchor absent	$259$	$0.4517$	—

Table 26: Mid-category top-5 on real product names, cross-tabulated, each cell against its own matched null. The null permutes labels within the cell , so it preserves that cell’s label composition;

p

is the fraction of

400

permutations reaching the observed value. Source experiments/real-ko-commerce/p1_stats.json .

Term	No fixed effects	$+$ source-category FE	Stratified
Lexical anchor	$+23.78$ pp ( $p{<}10^{-6}$ )	$\mathbf{+13.01}$ pp ( $p=0.003$ )	$+15.73$ pp (16/20 cat.)
Model code	$-35.47$ pp ( $p{<}10^{-8}$ )	$-4.01$ pp ( $p=0.51$ )	$+0.20$ pp (10/20 cat.)
Anchor $\times$ model code	$+18.74$ pp ( $p=0.005$ )	$+4.59$ pp ( $p=0.48$ )	—
$R^{2}$	$0.194$	$0.424$	—

Table 27: Correctness regressed on the two indicators,

n=1{,}174

, linear probability model with HC1 standard errors. Adding source-category fixed effects leaves the anchor effect standing and removes the model-code effect. The last column is an assumption-free check: the category-size-weighted within-category difference, over the categories that contain at least five queries on each side. Source p1_stats.json .

Same narrowed crosswalk, restricted to products absent from the training split
Crosswalk	L2 per label	top-5	null	lift
Hand-built (as reported above)	$2.55$	$0.7504$	$0.1843$	$+56.6$ pp
Kept only where the training split supports it	$1.60$	$0.6882$	$0.1576$	$+53.1$ pp
Single best-supported mid-category per label	$1.00$	$0.5571$	$0.1360$	$+42.1$ pp
Products unseen in training ( $n=837$ )	$1.60$	$0.7838$	$0.2376$	$+54.6$ pp

Table 28: Sensitivity of the result to how generous the crosswalk is. Lift is accuracy minus the mapping-preserving null, itself recomputed for each mapping. Source experiments/real-ko-commerce/map_sensitivity.json .

Arm	top-5	anchor present / absent	$\Delta$ vs deployed
BM25, leaf strings, char bigram	$0.6993$	$0.8055$ / $0.3243$	$-5.11$ pp $[-7.24,-2.89]$
Lexical-primary ( $w=50$ )	$0.6848$ $[0.658,0.710]$	$0.7650$ / $0.4015$	$-6.56$ pp $[-8.35,-4.77]$
Dense-only ( $w=0$ )	$0.7300$ $[0.704,0.756]$	$0.8164$ / $0.4247$	$-2.04$ pp $[-2.98,-1.19]$
Deployed hybrid ( $w=0.2$ )	$\mathbf{0.7504}$ $[0.726,0.775]$	$0.8350$ / $0.4517$	—
Teacher encoder, 768d / 421 MB	$0.7751$	$0.8175$ / $\mathbf{0.6255}$	$+2.47$ pp $[+0.09,+5.11]$
— same leaf construction as deployment	$0.7564$	$0.8175$ / $0.5405$	$+0.60$ pp $[-1.45,+2.73]$

Table 29: IR baselines on real Korean product names, mid-category top-5. The last column is a paired bootstrap against the deployed arm on the same queries. The first three arms are the shipping code with only the lexical weight changed; the teacher is jhgan/ko-sroberta-multitask , run on the same queries, leaf strings and crosswalk; the matched teacher is the same model fine-tuned on the student’s stage-4 (utterance, leaf) pairs with the same symmetric InfoNCE (§ 10.2 ). Sources experiments/real-ko-commerce/ir_baselines.json , teacher_arm.json and matched_teacher.json .

Ranking signal	100%	90%	80%	70%	60%
Score margin alone	$0.7504$	$0.7928$	$0.8296$	$0.8601$	$0.8977$
Anchor first, margin breaks ties	$0.7504$	$0.7938$	$0.8328$	$\mathbf{0.8723}$	$\mathbf{0.9048}$
$+$ penalize model-code form	$0.7504$	$0.8070$	$0.8317$	$0.8869$	$0.8807$

Table 30: Selective risk on real product names: accuracy when only the most confident

X\%

of queries are answered. Source experiments/real-ko-commerce/selective_risk.json .

Axis	Shipping	Previous	Diff	Significant
KLUE-STS ( $\rho$ )	$0.6661$	$0.6844$	$-0.02$	Yes
KLUE-YNAT (accuracy)	$0.6750$	$0.6871$	$-1.21$ pp	Yes
NSMC (accuracy)	$0.7548$	$0.7573$	$-0.25$ pp	No

Table 31: Shipping build ( stage6cov985_e10 ) versus the previous one ( stage5cap2k_e10 ), real-data axes. Source experiments/distill-ko/promote_gate_cov985_e10.json .

Artifact	gzip bytes	Notes
embedding.bin	$1{,}785{,}809$	$31{,}497\times 128$ @ 4bit, per-row scaling. raw $2{,}141{,}820$
catalog.bin	$491{,}922$	Ad catalog, $6{,}020$ items (synthetic), 4bit per-row
taxonomy.bin	$414{,}737$	$6{,}020$ leaves, 4bit per-row (§ 5.5 )
vocab.txt	$124{,}540$	Vocabulary of $31{,}497$
user_embedding.bin	$51{,}252$	User embedding, off by default
head.bin	$8{,}157$	Trained intent head

Table 32: L1 data budget measurements (gzip transfer bytes), shipping 4-bit artifacts. Source artifacts/ui-facts.json in the evidence bundle [ 46 ] .

Line	Measured	Cap	Share	Method
L1 data	$2{,}876{,}825$	$4{,}194{,}304$	$68.6\%$	gzip
L2 code	$49{,}979$	$1{,}048{,}576$	$4.8\%$	Native bytes left by the linker; development platform
Deployment payload (L1+L2)	$\mathbf{2{,}926{,}804}$	$\mathbf{3{,}145{,}728}$	$\mathbf{93.0\%}$	worst measured target
L3 resident	$2{,}352{,}740$	$8{,}388{,}608$	$28.0\%$	Resident bytes after unpacking

Table 33: Measurements for all three budget lines. L2 is the post-strip difference between an executable that calls the core and an empty executable (the native bytes the linker leaves behind).

Target	L2 code	L1 $+$ L2	Share of 3 MiB
aarch64-apple-darwin	$49{,}979$	$2{,}926{,}804$	$93.0\%$
aarch64-apple-ios	$49{,}603$	$2{,}926{,}428$	$93.0\%$
aarch64-linux-android	$\mathbf{65{,}827}$	$\mathbf{2{,}942{,}652}$	$\mathbf{93.5\%}$

Table 34: Deployment payload by build target. L1 is identical across targets; only the linked library code differs. All three targets now have a measured link. Source artifacts/code-size.json .

Configuration (2,960-leaf probe)	$p_{95}$
User embedding off (default)	$1.421$ ms
User embedding on, dense history	$1.763$ ms
Increase	$\mathbf{+0.342}$ ms

Table 35: Cost of switching the user embedding on, measured as a paired difference on one machine. The absolute values are not the shipping-path figure : this probe predates the taxonomy expansion and ran over

2{,}960

leaves, not

6{,}020

. What transfers is the difference. Source experiments/user-vector-gap/ue_latency_probe.json .

Device	SoC	$p_{50}$ ms	$p_{95}$ ms	$p_{95}$ range
iPhone 12 Pro ( iPhone13,3 ), iOS 26.4.2	A14 (2020)	$4.154$	$\mathbf{4.431}$	$4.2144$ – $4.5215$
iPhone 14 Pro Max ( iPhone15,3 ), iOS 26.6.2	A16 (2022)	$3.579$	$\mathbf{3.670}$	$3.6226$ – $3.7800$
Galaxy Tab A9+ ( SM-X210 ), Android 15	Snapdragon 695 (2021)	$4.991$	$\mathbf{5.080}$	$5.0511$ – $5.658$

Table 36: Tier-0 latency on two iOS handsets two SoC generations apart and one budget Android tablet.

p_{95}

is the median of nine launches and the range is those nine;

p_{50}

is the median-

p_{95}

launch’s own value, not a nine-run median.

512

-character input,

200

iterations after warm-up, the deployed

6{,}020

-leaf taxonomy. iOS thermal state nominal, low-power mode off. Each device was measured alone. Sources artifacts/ios-suite.json and artifacts/android-suite.json .

CPU	$p_{50}$ ms	$p_{95}$ ms	$p_{95}$ range
AMD EPYC 9355 32-Core Processor (64 threads)	$1.985$	$2.000$	$1.989$ – $3.085$
AMD Ryzen 7 7800X3D 8-Core Processor (16 threads)	$2.552$	$3.133$	$2.929$ – $3.368$
Intel XEON PLATINUM 8592V (256 threads)	$3.334$	$3.366$	$3.354$ – $3.549$

Table 37: Tier-0

p_{95}

on server CPUs, median of seven runs with the observed range. These are shared cluster nodes, so another tenant’s load can enter the measurement; they are observations, not floors. Source experiments/server-compare/cpu_latency.json .

#	Claim	Judge	Measured
C1	Deployment payload $\leq 3{,}145{,}728$	size_gate.py	$2{,}942{,}652$ B ( $93.5\%$ ; L1 $2{,}876{,}825$ $+$ L2 $65{,}827$ , worst of the three measured targets, Android; iOS gives $2{,}926{,}428$ B)
C2	$p_{95}<20$ ms	latency_gate.py	$\mathbf{4.431}$ ms (iPhone 12 Pro) and $\mathbf{3.670}$ ms (iPhone 14 Pro Max), § 10.6 ; $2.00$ – $3.37$ ms on three server CPUs; no development-Mac figure (§ 10.7 )
C3	No raw tokens in transmitted bytes	egress_gate.py	Canaries $7/7$ undetected; payload $79{,}968$ B of $307{,}200$
C5	Rounds converge despite dropout and latency, without masking applied	fl_round_gate.py	$50/50$ at $30\%$ dropout, BCE $0.6936\to 0.2125$
I1	wasm and native are bit-identical	parity_gate.py	max absolute diff $0.000$ over $1{,}408$ elements
C8	Campaign $\varepsilon$ within budget	dp_gate.py	Per-round $2.196$ ; campaign $1.542$ at $q=0.1$ (PLD, $\delta=10^{-10}$ )

Table 38: Gate summary. Each row is the named gate’s own output; the figures it measured are pinned in artifacts/gate-facts/ (one JSON per gate, written by the same call that prints the verdict).

† no ledger — see caption
Measurement	Training samples	Text accuracy	Behavior gain
Real logs (Amazon)	$4{,}988$	$0.5437$	$+12.83$ pp
Synthetic, templated utterances	$5{,}250$	$0.6789$	$+9.31$ pp
Synthetic, templated utterances, sample count reduced only	$1{,}187$	$0.5429$ †	$+3.02$ pp †
Synthetic, LLM-generated utterances	$1{,}187$	$0.7577$	$+0.52$ pp

Table 39: Four measurements of the same axis. Rows 1, 2 and 4 are backed by ledgers ( experiments/event-intent/real_results.json , results.json , fusion_llm_results.json ). Row 3 is not : the harness writes to a single path and that file now holds the

5{,}250

-sample run, so the

1{,}187

-sample run was overwritten and the script takes no sample-size argument. We mark it rather than delete it, and rather than re-run — a re-run would be a new measurement standing in a lost one’s place, which is the substitution this paper argues against.

	Synthetic utterances	Real reviews
Mean length (characters)	$8.9$	$39.4$
Mid-categories predicted	$255/256$	$214/256$
Entropy of the L2 distribution (bits)	$7.409$	$\mathbf{6.634}$
Share taken by the top 5 mid-categories	$9.8\%$	$\mathbf{19.1\%}$

Table 40: Prediction spread on synthetic utterances versus real commerce text. No ground truth is used; these are properties of the predicted distribution. Source experiments/real-ko-commerce/distribution_probe.json .

On-Device Inference

On-device inference runs machine-learning models locally on phones, wearables, microcontrollers, and edge hardware to reduce latency, energy use, and data sharing.

See all 189 papers

Jul 1, 2026

IntentTune: Using user demand and personalization to resolve "unknown" query intents for e-commerce search

Understanding user intent is fundamental to delivering relevant search results in e-commerce. However, substantial fraction of real-world queries are under-specified (e.g., "watch" or "shirt"), lacking explicit attributes such as gender or age group. This ambiguity poses a significant challenge for query intent detection models in e-commerce search systems, which must accurately infer latent user intent (e.g., age, gender) to support effective downstream retrieval. We introduce IntentTune, a framework for resolving ambiguous or under-specified query intents by leveraging either (1) user-specific behavioral signals including search history, browsing activity, and profile attributes or (2) population-level demand patterns aggregated across all users. Through experiments on real-world e-commerce data, we first demonstrate that population-level demand patterns alone are insufficient to reliably infer intent in under-specified queries. We then demonstrate that user-specific behavioral signals -- particularly prior search queries -- outperform both population-level statistics and static profile information for inferring gender, age group, product category, and size intent from underspecified queries.

Rachith Aiyappa, Ishita Khan, Chester Palen-Michel +4

eBay Inc., USA

E-Commerce Search Intent Classification User Modeling Information Retrieval

Ask PDF

Sep 4, 2026

Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation

Trade-up recommendation identifies higher-quality alternatives that preserve a customer's purchase intent while offering upgraded benefits. Large language models (LLMs) can reason about such distinctions, but applying them directly to hundreds of millions of product pairs is operationally impractical. We introduce a two-level framework that distills LLM reasoning into an efficient non-generative student and adapts its decision boundary to product-type-specific trade-up criteria. At Level 1, a retrieval-augmented few-shot LLM teacher generates structured relation labels and natural-language rationales. These rationales supervise a compact embedding-pair classifier through alignment and contrastive objectives; at inference, the student uses only two precomputed 768-dimensional product embeddings, with no LLM calls or text generation. On a fixed human-annotated benchmark of 8,352 pairs, a 15.5M-parameter four-class reasoning-distilled student achieves AUC 0.924 (95% CI [0.918, 0.929]), compared with 0.912 for the four-class label-only student. At Level 2, product-type test-time training (PT-TTT) uses few-shot demonstrations to optimize lightweight category-specific adapters over the frozen student. PT-TTT improves AUC from 0.924 to 0.941 and average precision from 0.920 to 0.940. On a 100K-pair proxy catalog, the distilled student on a single eight-GPU machine is approximately 5,000x faster and 10,000x lower in estimated cost than direct LLM inference.

Siliang Liu, Mohammad Ghasemi, Sapan Patel +1

Amazon Everyday Essentials Technologies

Reasoning Distillation Large Language Model-Based Recommendation Knowledge Distillation Test-Time Training

Ask PDF

Aug 17, 2026

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a buyer needs before a published number can be used: whether the retrieval backbone may be deployed commercially, and what it costs to build. On licensing: the field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0. Of the four leading MuSiQue systems we audit (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none says so. On performance: we measure thirteen embedders from eight makers on one identical MuSiQue harness with bootstrap confidence intervals throughout. Until mid-2026 there was a real commercial tax: the best commercially-licensed embedder trailed the anchor by 2.31 Recall@5 points (95% CI [0.91, 3.71], p=0.001). NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, has closed it: +0.24 at Recall@5 (95% CI [-0.94, +1.43], p=0.69), -0.58 at Recall@10 (p=0.28). It matches the anchor, does not beat it, and is the only entrant that is commercially licensed, free to self-host, and indistinguishable from the anchor; every other entrant meeting the first two conditions sits 5.2 to 14.6 points below. The durable finding is the paid-versus-free divide: API embedders charge per token on every re-index, self-hosted ones charge nothing. On cost: three of five audited systems (adding Microsoft's GraphRAG) do not disclose indexing cost, and the only published GraphRAG dollar figures span 11x inside one third-party paper (USD 2.30 vs USD 24.94 to index a 5.64 MB corpus once); extrapolated to 1 TB that undisclosed choice separates roughly USD 428K from $4.6M. Our cost model keeps one-time embedding apart from recurring answering: at 1 TB, embedding sits 7.5x-900x below graph construction, and a year of answering at 10,000 queries/day sits 350x or more below it.

Luis M. Sanchez, Kosrow Dehnad

Toryx

Dense Retrieval Text Embedding Evaluation IR Evaluation

Ask PDF

Abstract

On-Device Inference

Explore similar work

IntentTune: Using user demand and personalization to resolve "unknown" query intents for e-commerce search

Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks