cs.CLSep 6, 2026
SaveVisual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videos
Organizations: Virginia Tech · Amazon
Abstract
Existing approaches to visual attribute value extraction (AVE) primarily rely on static product images, failing to capture temporal cues, multi-angle views and fine-grained visual details. Directly applying video vision-language models (VLMs) to product AVE results in limited performance due to the lack of domain knowledge, and fine-tuning them requires extensive high-quality data and substantial computational resources. Thus, we propose visual search augmented chain-of-thought reasoning (ViS-CoT), a training-free, plug-and-play pipeline that can be easily applied to any open-source video VLM for video-to-text AVE in e-Commerce. Specifically, ViS-CoT employs visual clustering to identify representative frames, followed by visual search to retrieve semantically similar product knowledge that can enrich attribute cues. Next, an interleaved CoT reasoning module iteratively refines reasoning through visually-aligned auxiliary texts derived from captioning and automatic speech recognition. Finally, the integrated information guides the model toward accurate and fine-grained attribute predictions. Extensive experiments across 14 product categories on the VideoAVE dataset show that ViS-CoT consistently enhances multiple state-of-the-art video VLMs, achieving an average improvement of 17.91 percentage points in micro-F1.
Figures & tables
Figure 1: Comparison of product attribute value generation from multi-images vs. from videos.
Figure 2: The framework of ViS-CoT, which includes four stages. In stage (a), key frames are extracted from the raw product video through filtering and clustering. In stages (b) and (c), video and the key frames are processed to generate four types of textual information: (i) similar product information via visual search, (ii) image captions, (iii) refined transcripts from ASR, and (iv) interleaved CoT reasoning. In stage (d), all auxiliary textual information from stages (b) and (c) are integrated with the key frames to produce the final answer.
| Model | Appl. | Arts | Auto | Baby | Beauty | Clothes | Groc. | Indus. | Music | Patio | Pet | Phones | Sports | Toys | Gains |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Attribute-Conditioned Setting | |||||||||||||||
| VideoLLaVA | 15.96 | 11.68 | 13.26 | 12.53 | 10.97 | 19.73 | 10.16 | 14.04 | 15.16 | 12.93 | 15.09 | 13.02 | 14.56 | 15.26 | - |
| VideoLLaVA + ViS-CoT | 29.27 | 23.58 | 21.30 | 33.97 | 25.54 | 42.31 | 31.84 | 29.56 | 32.42 | 28.97 | 23.17 | 25.16 | 29.64 | 29.93 | ↑15.17 |
| VideoLLaMA3 | 33.33 | 28.72 | 29.22 | 33.44 | 29.20 | 35.93 | 25.97 | 31.35 | 31.84 | 29.56 | 29.49 | 30.34 | 31.89 | 33.10 | - |
| VideoLLaMA3 + ViS-CoT | 41.80 | 33.62 | 36.22 | 48.19 | 40.98 | 55.11 | 44.25 | 37.19 | 42.16 | 42.66 | 41.92 | 35.63 | 36.06 | 37.94 | ↑10.03 |
| InternVideo2.5 | 39.22 | 31.67 | 32.22 | 35.22 | 32.01 | 34.93 | 26.87 | 35.24 | 37.46 | 32.18 | 30.82 | 35.17 | 33.61 | 33.94 | - |
Table 1: Category-level fuzzy F1 (%) under the attribute-conditioned and generalized settings. Best results are highlighted in bold, and models with a light blue background denote the base model augmented with the ViS-CoT.
| Model | Brand | Color | Material | Country of Origin | Item Form | Power Source | Shape | Suggested Users | Pattern | Finish Type |
|---|---|---|---|---|---|---|---|---|---|---|
| VideoLLaVA | 6.49 | 19.62 | 15.12 | 15.61 | 13.02 | 2.31 | 22.89 | 2.28 | 21.85 | 10.68 |
| VideoLLaVA + ViS-CoT | 28.67 ↑22.18 | 22.44 ↑2.82 | 31.02 ↑15.90 | 21.34 ↑5.73 | 33.19 ↑20.17 | 41.64 ↑39.33 | 31.31 ↑8.42 | 42.55 ↑40.27 | 29.38 ↑7.53 | 16.65 ↑5.97 |
| VideoLLaMA3 | 43.27 | 32.76 | 28.34 | 29.68 | 19.82 | 8.82 | 34.70 | 1.68 | 27.64 | 14.52 |
| VideoLLaMA3 + ViS-CoT | 34.90 ↓8.37 | 28.84 ↓3.92 | 37.08 ↑8.74 | 47.06 ↑17.38 | 44.03 ↑24.21 | 58.83 ↑50.01 | 56.45 ↑21.75 | 45.57 ↑43.89 | 56.25 ↑28.61 | 32.42 ↑17.90 |
| InternVideo2.5 | 42.41 | 34.95 | 28.37 | 43.78 | 19.45 | 8.03 | 29.95 | 1.58 | 44.13 | 17.47 |
| InternVideo2.5 + ViS-CoT | 42.12 ↓0.29 | 39.71 ↑4.76 | 51.33 ↑22.96 | 75.21 ↑31.43 | 47.90 ↑28.45 | 70.69 ↑62.66 | 63.08 ↑33.13 | 80.21 ↑78.63 | 68.94 ↑24.81 | 40.02 ↑22.55 |
Table 2: Attribute-level fuzzy F1 (%) results. Top 10 frequent attributes are reported. Best results are highlighted in bold, and models with a light green background denote the base model augmented with the plug-and-play ViS-CoT.
Figure 3: Performance comparison (micro-F1%) of frames with random sampling, key frame clustering, visual search augmentation, CoT reasoning, ViS-CoT with search-enhanced interleaved CoT, and a full-size finetuned base model using training data.
Figure 4: Case study on a Cocoon organizer sleeve. Top: product video key frames. black border: original extracted frames; blue border: clustered/selected key frames. Left panel (Baseline): uniformly sampled video frames, without key-frame selection. Right panel (ViS-CoT): (1) clustered & cleaned key frames, (2) visual caption from the selected frames, (3) ASR transcript (speaker mentions brand, functions, and details), (4) visual-search neighbors providing attribute priors; Fusing (1)–(4) corrects Material Nylon and Pattern Solid; Closure remains biased to Zipper (gold: Magnetic).
Figure 5: Performance with different number of top- in visual search when using Qwen2.5-VL-7B as the base.
Figure 6: Performance with different cluster number in visual clustering on four categories of VideoAVE.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Latency (s) | % |
|---|---|---|
| Keyframe clustering (subtotal) | 2.40 | 48.1 |
| SigLIP embedding | 2.24 | (44.9) |
| K-means | 0.16 | (3.2) |
| Visual search (FAISS) | 0.11 | 2.2 |
| CoT Stage 1 (vision hypothesis) | 0.97 | 19.4 |
| CoT Stage 2 (grounded extraction) | 1.51 | 30.3 |
Table 3: Inference latency breakdown of ViS-CoT on a representative toy product. Indented gray rows break down the keyframe-clustering subtotal. Keyframe clustering is a one-time preprocessing step per video. Excluding it, per-query inference requires only 2.59 s vs. 1.31 s for the baseline.
| Condition | F1 (%) | F1 | s/product | F1 eff. |
|---|---|---|---|---|
| C0 Baseline | 33.4 | — | 1.6 | — |
| C1 +Visual Search | 48.3 | 2.8 | 12.3 pp/s | |
| C2 +CoT (w/o VS) | 33.5 | 3.2 | 0.1 pp/s | |
| C3 Full ViS-CoT | 58.5 | 3.3 | 14.3 pp/s |
Table 4: Per-condition inference time vs. F1 gain. F1 efficiency = F1 per additional second over C0. Full ViS-CoT delivers the best return on compute: each added second yields 14.3 pp of F1 improvement.
| Category | C0 | C1 (+VS) | C2 (+CoT) | Full |
|---|---|---|---|---|
| Arts | 31.6 | 43.1 ↑11.5 | 31.7 ↑0.1 | 49.1 ↑17.5 |
| Baby | 37.6 | 57.2 ↑19.6 | 39.6 ↑2.0 | 63.6 ↑26.0 |
| Clothing | 32.4 | 47.3 ↑14.9 | 33.3 ↑0.9 | 67.9 ↑35.5 |
| Grocery | 31.0 | 47.9 ↑16.9 | 31.8 ↑0.8 | 56.5 ↑25.5 |
| Patio | 33.1 | 50.0 ↑16.9 | 32.3 ↓0.8 | 61.3 ↑28.2 |
| Pet | 30.9 | 40.2 ↑9.3 | 28.6 ↓2.3 | 53.0 ↑22.1 |
Table 5: Component ablation (micro-F1%, / vs. C0). C0: base VLM with images and attributes only; C1: C0 + Visual Search (RAG); C2: C0 + interleaved CoT only, no retrieval; Full (C3): complete ViS-CoT. Subscripts show absolute change from C0.
| Category | Full | w/o Cap. | w/o Trans. | w/o RAG |
|---|---|---|---|---|
| Arts | 49.1 | 39.5 ↓9.6 | 39.4 ↓9.7 | 37.9 ↓11.2 |
| Baby | 63.6 | 57.5 ↓6.1 | 58.0 ↓5.6 | 54.5 ↓9.1 |
| Clothing | 67.9 | 61.7 ↓6.2 | 61.4 ↓6.5 | 59.8 ↓8.1 |
| Grocery | 56.5 | 48.1 ↓8.4 | 49.2 ↓7.3 | 46.3 ↓10.2 |
| Patio | 61.3 | 54.4 ↓6.9 | 54.4 ↓6.9 | 51.8 ↓9.5 |
| Pet | 53.0 | 47.8 ↓5.2 | 48.0 ↓5.0 | 46.5 ↓6.5 |
Table 6: Modality knock-out ablation (micro-F1%, drop from Full). Cap.=visual captions; Trans.=ASR transcripts; RAG=visual search. All three modalities are necessary; removing any one degrades performance across all categories.
| Condition | Avg F1 | vs. C1 |
|---|---|---|
| C0 Baseline | 33.4 | |
| C1 +Visual Search | 48.3 | — |
| Single-pass | 46.5 | |
| Full ViS-CoT | 58.5 |
Table 7: Single-pass vs. interleaved reasoning (micro-F1%, average over seven categories). C0: images only; C1: retrieval direct extract; Single-pass: retrieval + captions + ASR in one flat prompt (no Stage-1); Full ViS-CoT: same evidence, staged design. Adding all auxiliary evidence in one prompt actually underperforms visual search alone by 1.8 pp, while the staged design improves over C1 by +10.2 pp.
| Metric | Full set | Clean subset | |
|---|---|---|---|
| C0 baseline (F1%, raw) | 28.0 | 26.4 | |
| C1 Visual Search gain | pp | pp | pp |
Table 8: Visual search performance on the full evaluation set vs. the clean subset (test products whose product_id does not appear in training). The +13.3 pp clean-subset gain is only 1.6 pp ( 11% relative) below the +14.9 pp full-set gain, confirming that the improvement is not driven by near-duplicate label leakage.
| Category | GPT-4.1 | Qwen2.5-7B |
|---|---|---|
| Arts | 49.1 | 49.3 ↑0.2 |
| Baby | 63.6 | 63.0 ↓0.6 |
| Clothing | 67.9 | 67.7 ↓0.2 |
| Grocery | 56.5 | 55.8 ↓0.7 |
| Patio | 61.3 | 61.2 ↓0.1 |
| Pet | 53.0 | 53.1 ↑0.1 |
Table 9: End-to-end ViS-CoT micro-F1 (%) with GPT-4.1 vs. open-source Qwen2.5-7B for transcript summarization and generation. Subscripts show change vs. GPT-4.1. The pp average gap confirms that ViS-CoT’s gains do not depend on proprietary APIs.