On modern e-commerce stores, customers consume ordered slates of heterogeneous product media, such as images, videos, and 3D renders, before making purchase decisions. Existing media-ranking systems often optimize myopic engagement proxies such as clicks or dwell time, even though product media assets are cooperative informational components of the same item that together help customers find the information they need through sequential interaction. We present Sequential Multimodal Evidence Optimization (SMEO), a two-stage utility-guided framework for customer-oriented media sequencing. SMEO first learns a trajectory utility model from consumed media prefixes to estimate how ordered evidence helps customers reach a purchase decision, while mitigating position-bias and variable-depth imbalance in logged data. Recognizing that customer attention is a limited resource, it then trains an autoregressive ranking policy with survival-weighted reward-to-go that prioritizes the most decision-relevant information early, so customers can find what they need with less effort. By decoupling utility learning from policy optimization, SMEO enables stable offline learning from biased logs and post-hoc media attribution without explicit media-level labels. Evaluated offline on large-scale e-commerce sessions using doubly robust off-policy estimation, SMEO improves estimated conversion by 5.5% and helps customers reach a purchase decision with 15% fewer swipes than existing baselines.
Figures & tables
Figure 1. (Left) Product Detail page on Mobile. (Right) Suboptimal ordering may hide persuasive evidence until after customer drop-off, while ideal ordering front-loads decision-enabling media to support earlier conversion.
Figure 2. Complete end-to-end SMEO Framework
Method
Δ SNIPS-CVR ↑
Δ DR-CVR ↑
Δ MSC ↓
Heuristic (Rule-based)
–
–
–
Engagement(CTR)
+0.7±0.5%
+0.4±0.3%
−0.8±0.5%
CLTR-CVR
+1.5±0.7%
+1.1±0.5%
−3.6±0.8%
Listwise-MM
+3.2±0.6%
+2.7±0.4%
−8.5±1.2%
Seq2Slate-Adapt
+4.1±0.5%
+3.5±0.4%
−10.8±1.3%
SMEO
+6.1±0.6%
+5.4±0.3%
−15.3±1.5%
Table 1. For confidentiality, all metrics are reported as relative change against the heuristic baseline. Values include 95% confidence intervals. Higher is better for SNIPS-CVR, DR-CVR; lower is better for MSC.
Figure 3. (a) With the main product image fixed at slot 1, SMEO moves high-information assets earlier. Demo videos for coffee makers and nutrition labels for supplements produce larger LOO contribution when surfaced near the front. (b) For sneakers, near-identical views drive Δt=U(π≤t)−U(π≤t−1) toward zero after the first image, while diverse assets sustain marginal utility by adding complementary evidence.
Variant
Δ SNIPS-CVR ↑
Δ DR-CVR ↑
Δ MSC ↓
SMEO full
–
–
–
w/o PAL
−1.2±0.4%
−1.0±0.3%
+2.1±0.7%
PAL tower kept in U
−1.8±0.5%
−1.5±0.4%
+3.4±0.8%
w/o survival weighting
−2.3±0.5%
−1.9±0.3%
+4.8±0.9%
w/o interaction masking
−0.8±0.4%
−0.7±0.3%
+1.5±0.6%
Table 2. Ablation study of SMEO components. Values are relative to full SMEO.
Figure 4. Evidence accumulation curves across prefix depth.
Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
E-commerce videos are information-dense and frequently compared by consumers evaluating products and merchants assessing marketing strategies. However, existing multimodal models mainly focus on single-video understanding and have limited ability to compare information across videos. We introduce AdsCVR, the first e-commerce cross-video reasoning benchmark, containing 2,483 videos and 6,110 question-answer pairs across six reasoning dimensions. Cross- video reasoning requires models to locate fine-grained evidence among many redundant frames and integrate visual details, speech, and on-screen text. We therefore propose AdSeek, an agentic framework that dynamically selects visual and audio tools during multi-turn exploration, replacing static uniform sampling with active evidence acquisition. To address the sparse credit assignment of reinforcement learning, we develop an offline trajectory rectification mechanism that identifies reasoning errors and missing multimodal evidence in RL-generated trajectories. The corrected trajectories provide supervised fine-tuning signals that reduce biases learned during RL. This mechanism supports a rectified bootstrapping pipeline in which initial RL exposes reasoning bottlenecks, supervised fine-tuning corrects them, and a final RL stage further improves the policy. AdSeek achieves 74.30 percent accuracy on the AdsCVR test split, outperforming its Qwen3-VL-8B-Instruct backbone by 27.90 percentage points. It also generalizes to the open- domain CrossVid benchmark, demonstrating effective active evidence gathering.
Preference-based alignment objectives have been widely adopted, from RLHF-style pairwise learning in large language models to emerging applications in recommender systems. Yet, existing work rarely examines how Direct Preference Optimization (DPO) behaves under implicit feedback, where unobserved items are not reliable negatives. We conduct systematic experiments on multimodal sequential recommendation to compare common negative-selection strategies and their interaction with DPO training. Our central finding is that a simple modification, replacing deterministic hard negatives with stochastic sampling from a dynamic top-K candidate pool, consistently improves ranking performance. We attribute its effectiveness to two factors: (1) reducing erroneous suppressive gradients caused by false negatives, and (2) retaining informative hard signals while smoothing optimization via controlled stochasticity. With an optional sparse Mixture-of-Experts encoder for efficient capacity scaling, RoDPO achieves up to 5.25% NDCG@5 on three Amazon benchmarks, with nearly unchanged inference cost.