cs.CVNov 8, 2026

Query-Only Prompt-Agnostic Attacks on Visual In-Context Learning

Authors: Xiao Hu, Muhammad Ibrahim, Hao Wen, Pin-Yu Chen, Jianliang Wu, Antonio Bianchi

Organizations: Purdue University · Georgia Institute of Technology · Independent Researcher · IBM Research · 

Abstract

Visual In-Context Learning (VICL) enables promptable segmentation by conditioning RGB predictions on visual demonstrations, making it attractive for auto-labeling, but its robustness under query-only access and unknown deployment prompts remains underexplored. We study query-only, prompt-agnostic targeted attacks where the attacker observes only returned RGB outputs, without access to model internals or deployment prompts. Treating prompt-conditioned RGB rendering as a distinct attack surface, we introduce two appearance-space attacks: Color Hijack (CH) drives foreground colors toward the background to suppress visibility, while Task Hijack (TH) collapses instance-distinguishing colors toward a shared color assignment. To transfer across unseen prompts, we propose a Self-Prompt Surrogate (SPS) constructed from the clean query and its output. In a paired, matched-budget SegGPT comparison, SPS outperforms the Random-Prompt Surrogate (RPS), reducing CH instance foreground retention from 0.918 to 0.155 and TH palette-aware tolerance distance from 0.345 to 0.225. Across four VICL models on COCO and HierText, CH suppresses foregrounds across semantic-, instance-, and text-style outputs, while TH collapses instance colors under held-out deployment prompts. After palette-based decoding, TH also reduces the decoded instance count and mask retention while increasing instance merges. Finally, at 30% corruption, CH-corrupted annotations from a publicly accessible VICL-based auto-labeling pipeline reduce downstream YOLO-seg mAP50 from 0.761 to 0.281.

Figures & tables

Explore similar work

CardsList
  1. Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

    Apr 4, 2026Tianci Luo, Haohao Pan, Jinpeng Wang +5In-Context LearningRetrieval-Augmented ICL

  2. PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

    Mar 19, 2026Tianci Luo, Jinpeng Wang, Shiyu Qin +5In-Context LearningFeature Fusion

  3. Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection

    Sep 3, 2026Sizhe Chen, Yu-Lin Tsai, Ivan Evtimov +4Visual Prompt Injection AttacksPrompt Injection Attacks