PhoneValue-Bench: A Multimodal Benchmark and Datasets for Consumer Smartphone Recommerce Valuation
Organizations: School of Computing and Data Science, The University of Hong Kong · Weikean (Shenzhen) Trading Co., Ltd.
Abstract
As consumer smartphone recommerce scales, buyers must quote binding repurchase prices from seller-uploaded multi-view photos and diagnostic screenshots before devices reach the warehouse. Accurate remote valuation requires fusing noisy cosmetic evidence with identity, battery, lock, and channel risk from screenshots and staff forms, since an error in any modality changes both grade and payout. However, existing multimodal resources mainly target general VLM competence, screen reading, or factory defect detection, and largely overlook staff-aligned trade-in fields, dual appearance/staff ground truth, and executable valuation scorecards. This gap stems from two challenges: (1) the lack of large-scale real trade-in data with audited dual annotations and a compact ranking benchmark kept separate from training labels, and (2) the need to jointly reason over unordered multi-view cosmetics and diagnostic screens under operational deduction rules. To address these challenges, we propose PhoneValue-Bench, a dual-GT evaluation suite with tier-gated scorecards, together with PhoneValue-Corpus, a matched weakly labeled training set of over twenty thousand orders and hundreds of thousands of captioned images. We further present ValueAgent, a staged perception-fusion-valuation reference pipeline aligned with production inspection, plus an executable public scorer and submission schema. Experiments show that ValueAgent reaches a measured 84.1/100 on Bench-Full, with 97.0 appearance fill and 54.1 staff-field fill, establishing a practical dual-GT benchmark for multimodal recommerce valuation.
Figures & tables
| Dimension | DocVQA | ScreenQA | MMBench | RICO | SSGD | VisA | Kaputt | Ours |
|---|---|---|---|---|---|---|---|---|
| Large training / crawl set | ✗ | ✓ | ✓ | ✓ | ||||
| Compact audited dual-GT bench | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Multi-view device cosmetics | ✗ | ✗ | ✗ | ✗ | ✓ | |||
| Diagnostic screenshot fields | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | |
| Staff / order risk fields | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Tier-gated valuation scorecard | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| # | Stage | Operation |
|---|---|---|
| In: images , metadata Out: scorecard , grade , price | ||
| 1 | Preprocess | (six views) |
| 2 | Object detection | |
| 3 | Scene compliance | if : flag / reject |
| 4 | VLM (About) | |
| 5 | VLM (diagnostic) | |
| System | Det | VLM | Fuse | Score 100 | Y | S |
|---|---|---|---|---|---|---|
| ValueAgent (full) | ✓ | ✓ | ✓ | 84.1 | 97.0 | 54.1 |
| w/o Perception-Det | ✗ | ✓ | ✓ | 68.8 | 75.7 | 52.8 |
| w/o Perception-VLM | ✓ | ✗ | ✓ | 79.3 | 97.0 | 38.2 |
| w/o Fusion / policy | ✓ | ✓ | ✗ | 80.5 | 94.2 | 48.6 |
| Det-only (no VLM/Fuse) | ✓ | ✗ | ✗ | 74.4 | 97.0 | 22.0 |
| Gemini-2.5-Pro | ✗ | ✗ | ✗ | 69.9 | 78.0 | 51.0 |
| VA | Agree (%) | |||
| Item | Max | mean | VA | Direct-VLM |
| Y01 view coverage | 7 | 7.00 | 100.0 | 55.0 |
| Y02 class counts | 4 | 3.99 | 99.8 | 61.0 |
| Y03 dark-mode | 3 | 3.00 | 100.0 | 64.0 |
| Y04–Y07 defect F1 | 31 | 31.00 | 100.0 | 48.0 |
| Y08 grade agreement | 6 | 4.48 | 74.7 | 42.0 |
| Variant | Score 100 | Y fill | S fill |
|---|---|---|---|
| ValueAgent (full) | 84.1 | 97.0 | 54.1 |
| w/o Stage-3 scene gate | 82.7 | 95.1 | 53.8 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Name | Max | Comparison rule (vs. GT-Y) | Inactive if | Pass / fail example |
|---|---|---|---|---|---|
| Y01 | View coverage | 7 | Award over six canonical views | Order has 3 mandatory views after Step 2 | Pass: 6/6 views. Fail: missing Left_Frame |
| Y02 | Class counts | 4 | per active defect class full; else proportional | No defect classes annotated | Pass: 3 vs. 3 scratches. Fail: 0 vs. 11 |
| Y03 | Dark-mode / screen | 3 | Exact match on {on, off, dark} screen state from Front_A/B | No front screen photo | Pass: dark predicted=GT. Fail: on vs. off |
| Y04 | Side defect F1 | 8 | Box-level F1 at IoU on Left/Right_Frame defects | Both side views missing | Pass: F1 0.8. Fail: F1 0.2 |
| Y05 | Front defect F1 | 8 | Same as Y04 on Front_A/B | No front photo | Pass: crack box matched. Fail: miss hairline |
| Y06 | Back defect F1 | 8 | Same as Y04 on Back | No back photo | Pass: paint-loss matched. Fail: clutter FP |
| ID | Name | Max | Comparison rule (vs. GT-S) | Inactive if | Pass / fail example |
|---|---|---|---|---|---|
| S01 | Model identity | 3 | Normalized string match (case/space-insensitive; “iPhone13Pro” “iPhone 13 Pro”) | Staff model blank | Pass: 13 Pro. Fail: 13 vs. 13 Pro Max |
| S02 | Capacity | 2 | Exact match after GB/TB normalize | Staff capacity blank | Pass: 256GB. Fail: 128 vs. 256 |
| S03 | IMEI / serial | 1 | Last-4 digit match after mask; full match if unmasked GT | No IMEI in staff or screenshots | Pass: last-4 8399. Fail: wrong last-4 |
| S04 | Cosmetic grade | 6 | Exact staff string match (intentionally strict) | Staff grade blank | Pass: exact. Fail: warehouse lexicon vs. enum paraphrase |
| S05 | Grade ceiling | 3 | Predicted ceiling staff ceiling after A–E map (Table 8 ) | Staff ceiling blank | Pass: B B. Fail: A when staff C |
| S06 | Body cosmetic flag | 1 | Boolean match (true/false) | Flag not in staff form | Pass: both true. Fail: miss chip |
| GT-Y ( ) | Detector / Y08 bucket | Dashboard (A–E) |
|---|---|---|
| mint | mint | A (mint) |
| minor wear | minor | B (minor) |
| water ingress | major | C (major) |
| crack | crack | D (crack) |
| scratch (severe) | scrap | E (scrap) |
| Step | Name | Input output / gate |
|---|---|---|
| 1 | Collect Data | Raw orders (multi-view photos + diagnostic screenshots) 21,550 image-bearing orders |
| 2 | Preprocessing | Anonymize PII/IMEI; Pass/Quarantine on missing views, incomplete staff text, unreadable screenshots |
| 3 | Create Labels | Dual GT: GT-Y (mint, water ingress, minor wear, crack, scratch) + GT-S (model, capacity, battery, lock) |
| 4 | Build Corpus | PhoneValue-Corpus: 278,748 captioned images, weak YOLO exports, staff sidecars (train only) |
| 5 | Build Benchmark | PhoneValue-Bench: Bench-Check ( ) + Bench-Full ( , Tier-B, ) |
| # | Stage | Operation | Output / notes |
|---|---|---|---|
| 1 | Preprocess | Six canonical views | |
| 2 | Object detection | mint/minor/major/crack/scrap | |
| 3 | Scene compliance | if : reject/flag | Outdoor flag |
| 4 | VLM (About) | Model, capacity, lock | |
| 5 | VLM (diagnostic) | Battery / health | |
| – | Fusion + KB | Apply R1–R12 |
| ID | Name | Rule |
|---|---|---|
| R1 | Screenshot identity override | If is non-empty and disagrees with view-label model, keep screenshot model for S01. |
| R2 | Capacity screenshot priority | overrides metadata when both present and disagree. |
| R3 | Worst-region ceiling | in scrap crack major minor mint order. |
| R4 | Water S08+S10 | If any of {detector water-class, moisture, staff/repair water lexicon} fires: set S08=true; add water_severe to S10 blockers when severity major. |
| R5 | Activation lock | If About screen or KB flags activation/carrier lock: S10 adds activation_lock ; may force no-quote. |
| R6 | Channel inactive | If staff channel field blank: mark S09 inactive ( ); do not invent channel from UI language alone. |
| View | Style | English caption (Corpus) |
|---|---|---|
| Computer_screenshot | en_narrative | This About screen capture displays the software-identifiable information for an iPhone 13 Pro with 128GB storage and IMEI 3599***8399, sourced from an unlocked other country/region channel. The YOLO model classifies the image as a Computer_screenshot (0.95 confidence) with no detected defects. The staff assessment notes severe external damage, while the order-level YOLO condition hint indicates severe internal damage. |
| Right_Frame | en_narrative | This right side frame photo of the iPhone 14 Pro Max (128GB, IMEI 3582***8862) shows the device’s side profile. YOLO weak labels detect three minor side scratches and one side dent; staff assessment of heavy wear aligns with the YOLO condition hint. |
| Front_A | en_narrative | This front black-screen photo of an iPhone 6S Plus (64GB) shows cracked/delaminated screen cosmetics. YOLO reports Front_A with no active defect counts (face grade Normal), contrasting staff severe external damage. |
| Bottom_Frame | en_concise | Bottom screw-hole photo of an iPhone 13 Pro Max (256GB, heavy wear). YOLO detects 11 side scratches and 2 side dents. |
| Left_Frame | en_narrative | Left side frame of an iPhone 13 (128GB) with no YOLO defects; staff “excellent” aligns with the condition hint. |
| Front_B | en_technical | Front_B of an iPhone 13 Pro Max (256GB, IMEI 3579***1099). YOLO stage-2 identifies 21 screen scratches; staff grade heavy wear. |
| Field | Type | Example / notes |
|---|---|---|
| order_id | string | PV-FULL-0127 |
| views_present | string[] | Six canonical view names when present |
| defects | object[] | {view, cls, bbox} boxes for Y04–Y07 |
| detector_grade | enum | { mint , minor , major , crack , scrap } |
| model / capacity | string | Normalized for S01/S02 |
| imei_last4 | string | Masked last-4; null if unknown (do not invent) |
| Field | Example | Notes |
|---|---|---|
| y / y_max | 58.0 / 73 | Active sum / Tier-B max |
| score_100 | 79.5 | Tier-B fill formula |
| y_fill / s_fill | 94.1 / 45.5 | Appearance / staff (%) |
| item_scores | Y01=7.0, … | Points or inactive |
| gates | Y09=0, S11=0, … | Tier-B mask |
| Step | Component | Setting / rule |
|---|---|---|
| 1 | Input | All order images + fixed English schema prompt |
| 2 | Decoding | Temperature ; single-turn; no tools/RAG |
| 3 | Parse | JSON scorecard fields; malformed zeros |
| 4 | Scoring | Identical Tier-B gates and |
| 5 | Exclusions | No detector crops, fusion, or price table |
| # | Instruction line |
|---|---|
| 1 | You are scoring a used-phone trade-in order. |
| 2 | Given all images, output one JSON object. |
| 3 | Keys: views_present; defects[{view, cls, bbox}]. |
| 4 | detector_grade {mint, minor, major, crack, scrap}. |
| 5 | Also: model, capacity, imei_last4, staff_grade_string. |
| 6 | grade_ceiling {A, B, C, D, E}. |
| # | Instruction line |
|---|---|
| 1 | Read the About/settings or diagnostic screenshot only. |
| 2 | Extract: model, capacity, IMEI (mask middle). |
| 3 | Also extract: battery_health, lock_status, channel. |
| 4 | Return schema JSON; use null if unreadable. |
| Variant | Input | Stage-1 crop? | Failure policy |
|---|---|---|---|
| Direct-VLM (default) | All raw images | No | Malformed JSON 0 |
| VLM+boxes | Raw images + YOLO boxes as text | No | Missing boxes empty list |
| Crop-then-VLM | Six canonical crops only | Yes | Missing crop skip view |
| Multi-turn (2) | Turn1 JSON + Turn2 repair prompt | No | Keep Turn1 if Turn2 invalid |
| Tool-OCR | Images + OCR dump of screenshots | Optional | OCR empty null fields |
| Variant | Det | VLM | Fuse | Protocol |
|---|---|---|---|---|
| ValueAgent (full) | ✓ | ✓ | ✓ | Stages 1–6 + R1–R12 |
| w/o Perception-Det | ✗ | ✓ | ✓ | Skip Stages 1–2; raw images to Stages 4–5 |
| w/o Perception-VLM | ✓ | ✗ | ✓ | Zero About/diagnostic VLM JSON |
| w/o Fusion / policy | ✓ | ✓ | ✗ | Concatenate detector+VLM without R1–R12 |
| Det-only | ✓ | ✗ | ✗ | Stages 1–2 only; map to Y items |
| Direct-VLM | ✗ | ✗ | ✗ | Bag-of-images schema prompt (Table 15 ) |
| Slice | Grade 1 | Price 10% | Price 20% |
|---|---|---|---|
| All (excl. no-quote) | 78.0 | 71.0 | 84.0 |
| Mint / excellent | 86.2 | 79.4 | 91.1 |
| Minor wear | 81.5 | 74.2 | 87.3 |
| Heavy / major | 72.4 | 65.8 | 79.6 |
| Crack / scrap | 64.1 | 58.0 | 71.2 |
| Slice | Mean [95% CI] | |
|---|---|---|
| Bench-Full (all) | 335 | 84.1 [82.7, 85.4] |
| Cosmetic: mint / excellent | 48 | 91.2 [88.9, 93.1] |
| Cosmetic: minor wear | 142 | 86.4 [84.6, 88.0] |
| Cosmetic: heavy / major | 97 | 80.1 [77.8, 82.3] |
| Cosmetic: crack / scrap | 48 | 74.6 [71.2, 77.9] |
| Family: iPhone 11–13 | 156 | 85.0 [83.1, 86.7] |
| Attribute | Corpus | Bench-Check | Bench-Full |
|---|---|---|---|
| Orders | 21,550 | 500 | 335 |
| Images | 278,748 | – | – |
| Tier-A / B / C | 21,122 / 428 / 2 | 0 / 500 / 0 | 0 / 335 / 0 |
| Has diagnostic screenshot | 89.4% | 92.0% | 94.3% |
| Has 1 side view | 91.2% | 88.4% | 96.1% |
| Cosmetic: minor wear | 7,928 (36.8%) | 168 (33.6%) | 142 (42.4%) |
| # | Type | Observation | GT vs. pred | Attributed stage |
|---|---|---|---|---|
| 1 | S04 lexicon | Staff warehouse grade string vs. enum paraphrase | S04: 0 vs. expected 6 | MapToScorecard / schema |
| 2 | Missed side scratch | Left_Frame hairline ignored | Y04 F1 drop | Stage 2 detector |
| 3 | Outdoor false defect | Street clutter scratch FP | Y06 FP | Stage 3 (skipped) |
| 4 | IMEI OCR | Last-4 misread on glare | S03 fail | Stage 4 VLM |
| 5 | Blocker miss | Activation lock not flagged | S10 miss | Stage 4 + R5 |
| 6 | Water vs. mint | Photos mint; repair text water | S08/S10 after R9 | Fusion R9 |
| Check | Result |
| (order IDs) | 0 |
| (order IDs) | 0 |
| (order IDs) | 0 |
| Step 2 Pass / Quarantine (of image-bearing pool) | 78.4% / 21.6% |
| Quarantine: missing mandatory views | 11.2% |
| Quarantine: incomplete staff text | 6.1% |
| Severity | Y-fill | S-fill | ||
|---|---|---|---|---|
| Mint / excellent | 48 | 99.0 | 62.5 | 91.2 |
| Minor wear | 142 | 97.8 | 57.5 | 86.4 |
| Heavy / major | 97 | 96.2 | 50.5 | 80.1 |
| Crack / scrap | 48 | 93.8 | 42.8 | 74.6 |
| All (weighted) | 335 | 97.0 | 54.1 | 84.1 |
| Variant | Active | Score | Primary hit | |
|---|---|---|---|---|
| w/o Perception-Det | VLM+Fuse | 68.8 | Y01, Y04–Y07 | |
| w/o Perception-VLM | Det+Fuse | 79.3 | S01–S03, S10 | |
| w/o Fusion / policy | Det+VLM | 80.5 | S02/S08/S09 conflicts | |
| Det-only | Det only | 74.4 | S-fill 22.0 | |
| Direct-VLM (best) | none | 69.9 | Y 78, weak crops | |
| w/o Stage-3 | full pipe | 82.7 | outdoor Y06 FPs |
| Pair | |
|---|---|
| vs. grade 1 | 0.61 |
| vs. price 10% | 0.48 |
| vs. price 20% | 0.55 |
| Y-fill vs. grade 1 | 0.29 |
| S-fill vs. grade 1 | 0.67 |
| Y-fill vs. S-fill | 0.18 |
| Item | VA | DV | |
|---|---|---|---|
| Appearance items | |||
| Y01 | 100.0 | 55.0 | +45.0 |
| Y02 | 99.8 | 61.0 | +38.8 |
| Y03 | 100.0 | 64.0 | +36.0 |
| Y04–Y07 (pooled) | 100.0 | 48.0 | +52.0 |
| Y08 | 74.7 | 42.0 | +32.7 |
| Protocol element | What it enables | Ours |
|---|---|---|
| Dual GT (GT-Y + GT-S) | Separates cosmetic audit from payout fields | ✓ |
| Tier-gated | Fixed denominator across submitters | ✓ |
| Public scorer + JSON schema | Bit-identical leaderboard arithmetic | ✓ |
| Corpus Bench IDs | Training without ranking leakage | ✓ |
| Det/VLM/Fuse factor table | Comparable ablations vs. Direct-VLM | ✓ |
| Off-leaderboard price DSL | Business audits without leakage | ✓ |