Organizations: Shenzhen International Graduate School, Tsinghua University · The Chinese University of Hong Kong, Hong Kong SAR · McGill University · The Hong Kong University of Science and Technology, Guangzhou · Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute, China
Video-LLMs are increasingly deployed in safety-critical applications but are vulnerable to Energy-Latency Attacks (ELAs) that exhaust computational resources. Current image-centric methods fail because temporal aggregation mechanisms dilute individual frame perturbations. Additionally, real-time demands make instance-wise optimization impractical for continuous video streams. We introduce VidDoS, which is the first universal ELA framework tailored for Video-LLMs. Our method leverages universal optimization to create instance-agnostic triggers that require no inference-time gradient calculation. We achieve this through masked teacher forcing to steer models toward expensive target sequences, combined with a refusal penalty and early-termination suppression to override conciseness priors. Testing across three mainstream Video-LLMs and three video datasets, which include video question answering and autonomous driving scenarios, shows extreme degradation. VidDoS induces a token expansion of more than 205× and inflates the inference latency by more than 15× relative to clean baselines. Simulations of real-time autonomous driving streams further reveal that this induced latency leads to critical safety violations. We urge the community to recognize and mitigate these high-hazard ELA in Video-LLMs.
Figures & tables
Figure 1: Overview of VidDoS . The framework involves: (1) Defining the attack goal; (2) Temporal injection of a universal patch; (3) Steering decoder trajectory via joint losses (Masked Teacher Forcing, Refusal Penalty, and ETS); (4) Offline patch optimization via Sign-PGD; and (5) Real-time latency induction.
Dataset
Victim Model
Clean (No Attack)
Attack Method
Adversarial Performance
Tokens
Latency (s)
Adv Tokens
Ratio ( ↑ )
Adv Lat. (s)
Overhead ( ↑ )
BDDX
LLaVA-NeXT-Video-7B
28.40
1.12
Random Noise
31.35
5.15
1.22
-0.05
Verbose Images
31.00
5.03
1.22
-0.05
NICGSlowDown
32.50
1.74
1.19
0.07
VidDoS (Ours)
333.50
67.13
9.38
8.25
Qwen3-VL-4B-Instruct
2.00
0.16
Random Noise
2.00
1.00
0.15
0.00
Table 1: Quantitative comparison of attack methods. Best results are in bold .
Figure 2: Cumulative latency under VidDoS attack in video streaming scenario.
Figure 3: (a) Cross-dataset transfer. Each entry reports the average output length when a model trained on the source dataset (row) is evaluated on the target dataset (column). (b) Length distribution comparison. For each dataset, output length distributions of the original model and under VidDoS are compared.
Temp ( T )
Adv Tokens
Overhead (s)
Expansion Ratio
0.0 (Greedy)
487.25
19.21
243.62 ×
0.2
487.50
19.15
243.75 ×
0.5
487.30
19.53
243.65 ×
0.7 (Default)
511.40
20.20
255.70 ×
1.0
511.40
20.60
255.70 ×
1.2
511.40
20.39
255.70 ×
Table 3: Impact of decoding temperature on attack effectiveness (Qwen3-VL on BDDX) . All results use the same universal patch optimized with T=0.0 .
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.
The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8× more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.
Large vision-language models (LVLMs) have demonstrated strong performance in open-ended video understanding, yet they remain prone to fluent responses unsupported by video evidence. Existing training-free methods typically apply a globally fixed visual intervention or construct a contrastive branch through input perturbation. The former cannot accommodate video-dependent fusion paths, while the latter can be compensated by cross-frame redundancy. We therefore propose Video-Adaptive Debiasing via Evidence Reweighting (VADER), a training-free framework with two complementary modules. Visual Focus Reallocation (VFR) automatically instantiates an intervention policy for each video-question input: it diagnoses layer-wise visual-to-text evidence flow, determines where to intervene, and derives how strongly to reallocate pre-softmax attention from system-token to video-token blocks. Selective Evidence Erasure (SEE) independently masks high-importance visual tokens in every frame, constructing a prior-biased branch that is difficult to compensate through neighboring frames. Contrastive decoding then down-weights predictions that remain confident after selective evidence erasure. Across multiple VideoLLMs, VADER yields substantial improvements on event-level grounding and temporal consistency; on LLaVA-Video-7B, it reaches 72.60% accuracy on EventHallusion.
Dong Xing, Jiaxin Chen, Hang Yang +3
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences Changchun, Jilin 130033, China · University of Chinese Academy of Sciences, Beijing 100049, China · City University of Hong Kong, Hong Kong, China