Organizations increasingly deploy downloaded open-weight GUI agents. A backdoored agent passes acceptance tests on clean data and misbehaves only when a visual trigger appears on screen, turning an unvetted checkpoint into a supply-chain risk. However, existing sample-level detectors target image classifiers and CLIP-style encoders, leaving step-level backdoor detection for GUI agents unaddressed. On clean steps, the instruction drives the action; when an unreferenced screen region hijacks the step, the instruction loses control over the decision state or that region dictates it instead. We term this signature intent--evidence dominance inversion and propose IDEA (Intent-Dominance Evidence Audit) to detect backdoor activation at inference time for each step without trigger knowledge or attack data. Specifically, Intent Counterfactual Probing (ICP) measures how far redirected instructions move the decision state, and Region Causal Audit (RCA) quantifies how far ablating a candidate region moves it. A gated conformal score compares both displacements against clean steps at the same distance from the clean manifold, providing a finite-sample false-positive guarantee. Extensive experiments on Qwen2.5-VL-3B across five attack families show that IDEA consistently detects backdoor activation under a controlled false-positive rate and outperforms state-of-the-art detection methods, while localizing the trigger region for every alarm and adding only modest inference overhead.
Figures & tables
Figure 1: Intent–evidence dominance inversion. (a) Clean step: the action follows the instruction. (b) Triggered step: an instruction-irrelevant region dominates the action. (c) IDEA probes the instruction and audits screen regions; a flagged step returns region R⋆ .
Figure 2: Clean steps follow a presence law; triggered steps leave it on at least one side. Emax (top) and Sint (bottom) against presence, in log units (one seed). Gray: clean steps; line and band: fit on B1 and ±2 residual sd; color: followed triggered steps.
Figure 3: Overview of IDEA. A presence gate screens the decision state h0 . ICP redirects the instruction and RCA ablates proposed regions; both effects are compared with clean steps of the same presence, which gives the excess evidence u and the intent deficit v . A step that passes the gate with ρpc>Tα raises an alarm and returns the region R⋆ .
Table 1: Detection on the strict episode-disjoint split (20 seeds). TPR on followed triggered steps at 1% and 0.5% target FPR, and AUROC (AUC). AVG and Min: mean and worst family of the five TPR columns at each target. Best per column bold, second underlined.
Figure 4: Triggers leave the clean quadrant on different sides. Excess evidence u and intent deficit v (one seed). Color: followed triggered steps that pass the gate; light and dark gray: clean test steps that fail and pass it; shaded: alarm region ρpc>Tα . (a) Grounding. (b) Adaptive-Blend ( × : beyond the frame). (c) Patch, Popup and TypeText.
Figure 5: One alarm per family. Followed steps flagged at α=0.01 : trigger (red), returned region R⋆ (dashed teal), and the effects Ej of the M=4 proposals against Sint (dotted). (a–d) Localized triggers. (e) Full-screen blend.
α=0.01
α=0.005
Variant
Patch
Popup
Ground.
Adapt.
Type
AVG
AVG
IDEA (gated ρpc )
1.000
1.000
0.962
0.898
0.999
0.972
0.844
Remove one component
− presence gate
0.999
1.000
0.881
0.761
0.998
0.928
0.818
− conditioning (log-ratio)
1.000
1.000
0.950
0.513
0.999
0.893
0.814
Replace the either-side norm
Table 2: Every component matters, and only either-side rules hold on all five families. TPR on followed steps (strict split, 20 seeds); AVG over the five families. All rows share the forwards, fit and calibration. Best bold, second underlined.
Grounding
Adaptive-Blend
α
n
IDEA
Log-ratio
Screen
IDEA
Log-ratio
Screen
0.01
all ( ≈ 600)
0.962
0.950
0.543
0.898
0.573
0.891
0.005
all
0.702
0.761
0.276
0.569
0.313
0.713
0.01
400
0.952
0.929
0.542
0.883
0.564
0.854
0.01
200
0.891
0.905
0.514
0.715
0.432
0.845
0.01
100 †
0.869
0.882
0.501
0.782
0.537
0.805
Table 3: IDEA needs about 400 clean calibration steps and a 1% target. TPR on the two hard families (strict split, 20 seeds) at target α with n calibration steps. Best per family bold. † Certified at level 1/51 only.
AUROC among gate-passers
IDEA alarm rate
Family
ρpc
Screen
Emax
Log-ratio
Clean
Trig.
Patch
0.999
0.999
1.000
1.000
0.125
1.000
Popup
1.000
0.991
1.000
1.000
0.165
1.000
Grounding
0.987
0.791
0.892
0.984
0.151
0.978
Adaptive-Blend
0.967
0.954
0.989
0.769
0.149
0.912
TypeText
0.999
0.999
1.000
1.000
0.127
0.999
Table 4: Clean versus triggered steps among gate-passers (strict split, 20 seeds). AUROC between followed triggered and clean steps that pass the gate, and IDEA’s alarm rate in each group at α=0.01 .
Family
Hit
IoU > 0.5
mIoU
Patch
0.862
0.673
0.546
Popup
0.993
0.000
0.160
Grounding
0.910
0.138
0.318
TypeText
0.494
0.323
0.428
Adaptive-Blend
(1.000)
0.011
0.016
Table 5: R⋆ points at localized triggers but rarely matches their extent. Correct alarms at α=0.01 (strict split, 20 seeds). Hit: center of R⋆ inside the trigger box. Parentheses: trivial.
IDEA (gated ρpc )
Mahalanobis screen
Emax alone
α
Family
TPR
FPR T [CP95]
App-cluster CI
Worst seed
Log-ratio TPR
TPR
FPR T [CP95]
TPR
FPR T [CP95]
0.01
Patch
1.000
0.006 [0.005, 0.009]
[0.0036, 0.0096]
7/339
1.000
0.999
0.009 [0.006, 0.011]
1.000
0.009 [0.007, 0.012]
Popup
1.000
0.008 [0.006, 0.011]
[0.0053, 0.0121]
7/339
1.000
0.999
0.008 [0.006, 0.010]
1.000
0.008 [0.006, 0.011]
Grounding (point)
0.962
0.008 [0.006, 0.010]
[0.0042, 0.0111]
6/298
0.950
0.532
0.009 [0.007, 0.012]
0.288
0.010 [0.008, 0.013]
Grounding (bbox)
0.961
0.008 [0.006, 0.010]
[0.0042, 0.0111]
6/298
0.949
0.530
0.009 [0.007, 0.012]
0.290
0.010 [0.008, 0.013]
Adaptive-Blend
0.898
0.007 [0.005, 0.009]
[0.0038, 0.0109]
7/298
0.513
0.891
0.009 [0.006, 0.011]
0.919
0.009 [0.007, 0.012]
Table 6: FPR intervals on the strict split (20 seeds). TPR on followed steps, pooled FPR T [CP95], app-cluster bootstrap CI over 427 test apps, and worst seed as k/n ; log-ratio: unconditional ablation.
Family
Rule
N
α=0.01
min–max
α=0.005
Gate pass
Patch
point
2
1.000 ± 0.000
1.000–1.000
0.700 ± 0.458
1.000 ± 0.000
Popup
bbox
0
no ignored steps
Grounding
point
92
0.943 ± 0.045
0.848–0.989
0.687 ± 0.236
0.979 ± 0.037
Grounding
bbox
3
1.000 ± 0.000
1.000–1.000
0.667 ± 0.365
1.000 ± 0.000
Adaptive-Blend
point
19
0.547 ± 0.042
0.474–0.632
0.382 ± 0.178
0.608 ± 0.070
TypeText
point
142
1.000 ± 0.000
1.000–1.000
0.961 ± 0.102
1.000 ± 0.000
Table 7: Alarm rate on ignored steps (trigger present, attack failed) on the strict split. Mean ± sd over 20 seeds, min–max at α=0.01 , and gate pass rate. N : ignored steps among the 1000 triggered steps.
TPR@1% FPR on followed steps / held-out clean FPR with CP95
Detector
Patch
Popup
TypeText
Grounding
Adaptive-Blend
IDEA (gated ρpc )
1.000 /0.017
1.000 /0.019
1.000 /0.016
0.999 /0.017
0.983 /0.020
[.010,.027]
[.012,.030]
[.009,.026]
[.010,.027]
[.012,.031]
Log-ratio (ablation, gated ρ )
1.000 /0.011
1.000 /0.013
1.000 /0.017
0.999 /0.016
0.688/0.018
Mahalanobis screen (gate alone)
1.000 /0.011
1.000 /0.011
1.000 /0.009
0.877 /0.012
0.949 /0.009
One-class SVM (OC-SVM)
1.000 /0.014
1.000 /0.018
1.000 /0.018
0.498/0.011
0.882/0.011
Table 8: Secondary split (overlapping episodes, one seed). TPR at 1% target FPR on followed steps / measured held-out clean FPR [CP95]; best per column bold, second underlined; —: not run.
Earlier IDEA score (single split)
Mahalanobis screen (single split)
Victim × trigger
ASR
Alarm
Clean FPR
Followed
Alarm
Clean FPR
Followed
Clean-victim control
Clean × TypeText
0.000
0.004 [0.001, 0.010]
0.002 [0.000, 0.007]
0/0
0.011 [0.006, 0.020]
0.013 [0.007, 0.022]
0/0
Clean × Grounding
0.006
0.003 [0.001, 0.009]
0.002 [0.000, 0.007]
0/6
0.011 [0.006, 0.020]
0.013 [0.007, 0.022]
1/6
Clean × Adaptive-Blend
0.001
0.007 [0.003, 0.014]
0.002 [0.000, 0.007]
0/1
0.010 [0.005, 0.018]
0.013 [0.007, 0.022]
1/1
Foreign-overlay control
Table 9: Specificity controls. Top: superseded IDEA score and screen, single-split pipeline (one seed). Bottom: frozen rule on the strict split (20 seeds). Followed: alarms among steps taking the overlay’s target action. † Receiving victim’s own rule.
Steps
IDEA alarms
Screen alarms
Bin
C
T
C
T
C
T
1
4875
11
0
0
0
0
2
1293
3585
33
3002
5
128
3
33
4848
8
4762
11
1689
4
21
4855
1
4853
14
3206
5
33
4861
5
4860
30
4103
Table 10: Alarms at matched s0 (Grounding, α=0.01 , pooled over 20 seeds). Five common s0 bins per seed; alarms as counts over the clean (C) and followed triggered (T) steps of the bin.
Gate pass
IDEA alarm
Family
Seen
Unseen
Seen
Unseen
Patch
0.052
0.049
0.008
0.002
Popup
0.052
0.048
0.009
0.008
Grounding
0.052
0.040
0.009
0.002
Adaptive-Blend
0.048
0.044
0.007
0.006
TypeText
0.051
0.041
0.007
0.003
Table 11: Clean test steps from apps inside (Seen, 4946 steps) and outside (Unseen, 1309) the victim’s training pool (strict split, pooled over 20 seeds): gate pass rate and IDEA alarm rate at α=0.01 .
Patch victim
Grounding victim
Shift
Acc
Gate
IDEA
Scr.
Emax
Acc
Gate
IDEA
Scr.
Emax
None
0.860
0.049
0.009
0.012
0.009
0.860
0.042
0.008
0.009
0.007
No resizing
0.524
0.049
0.005
0.008
0.001
0.536
0.040
0.005
0.007
0.002
Noise
0.842
0.049
0.008
0.011
0.005
0.850
0.045
0.009
0.009
0.008
JPEG
0.852
0.047
0.009
0.011
0.008
0.850
0.042
0.009
0.010
0.008
Inversion
0.850
0.048
0.009
0.009
0.011
0.864
0.040
0.005
0.008
0.007
Table 12: Benign input shifts do not raise the alarm rate. 500 clean steps per set, thresholds from unshifted calibration ( α=0.01 , 20 seeds). Acc: victim action accuracy; Gate: gate pass rate; Scr.: Mahalanobis screen.
TPR@1% FPR / measured clean FPR
Detector
Patch
Popup
TypeText
Grounding
Adaptive-Blend
STRIP [ 13 ]
0.937/0.011
0.031/0.002
0.000/0.005
0.169/0.004
0.058/0.009
SCALE-UP [ 14 ]
0.000/0.000
0.000/0.000
0.000/0.000
0.000/0.000
0.000/0.000
Instruction contrast
0.000/0.000
0.000/0.000
0.000/0.000
0.000/0.000
0.000/0.000
VLM judge
0.000/0.003
0.481/0.005
0.028/0.004
0.003/0.002
0.000/0.002
AUROC (best TPR method)
0.988
0.976
0.647
0.868
0.686
Table 13: Per-victim protocol: extra baseline detectors. TPR@1% FPR / measured clean FPR on one fixed split, counting all triggered steps; bold: best TPR per column; last row: AUROC of that detector.
Figure 6: Valid redirects per step on the clean auxiliary set. Fraction of steps with at least k valid redirects under Eq. 1 , per split; the band marks k<2 , where Sint falls back to the clean median (red: its share).
Score
α
Ignored trig.
FPR T (CP95)
FPR, clean test
IDEA
0.01
73/1000
8/800 [0.004, 0.020]
8/1000
Log-ratio
0.01
53/1000
6/800 [0.003, 0.016]
5/1000
Screen
0.01
44/1000
10/800 [0.006, 0.023]
9/1000
Emax
0.01
53/1000
12/800 [0.008, 0.026]
7/1000
IDEA
0.005
56/1000
4/800 [0.001, 0.013]
1/1000
Log-ratio
0.005
43/1000
4/800 [0.001, 0.013]
5/1000
Table 14: Held-out blend, first collection (no resizing): alarm counts. No triggered step is followed, so no TPR is defined; FPR T on the held-out clean split [CP95] and on the 1000 clean test steps.
Variant
Patch
Popup
Ground.
Adapt.
TypeText
AVG
IDEA (gated ρpc )
0.978
0.996
0.702
0.569
0.976
0.844
− gate (ungated ρpc )
0.978
0.996
0.651
0.491
0.974
0.818
− conditioning (gated log-ratio)
1.000
1.000
0.761
0.313
0.997
0.814
u only (gated)
0.900
0.952
0.170
0.760
0.799
0.716
v only (gated)
0.882
0.959
0.774
0.137
0.907
0.732
u+v (gated)
1.000
1.000
0.810
0.253
0.997
0.812
Table 15: Component ablation at α=0.005 on the strict split. TPR on followed steps, mean over 20 seeds, and the five-family average. Bold: best per column. Ground.: Grounding; Adapt.: Adaptive-Blend.
M
Patch
Popup
Ground.
Adapt.
TypeText
AVG
1
1.000
1.000
0.965
0.827
1.000
0.958
2
1.000
1.000
0.963
0.886
0.999
0.970
3
1.000
1.000
0.963
0.894
0.999
0.971
4
1.000
1.000
0.962
0.898
0.999
0.972
Table 16: Proposal count. TPR on followed steps at α=0.01 with the candidate set cut to the top- M ranked proposals, replay on stored hidden states of the strict split, mean over 20 seeds. AVG over the five families.
Candidate set
Patch
Popup
Ground.
Adapt.
TypeText
top-4 attribution ( M=4 )
1.000
1.000
0.962
0.898
0.999
4×4 grid only
0.999
1.000
0.906
0.969
0.999
8×8 grid only
0.999
1.000
0.893
0.964
0.999
top-4 + 4×4
0.999
1.000
0.931
0.956
0.998
top-4 + 8×8
0.999
1.000
0.950
0.915
0.999
seed sd
0.004
0.000
0.082
0.091
0.004
Table 17: Candidate sets under the frozen rule. TPR on followed steps at α=0.01 , mean over 20 seeds, with the seed standard deviation and the pooled FPR T range over the five sets of each family. Ground.: Grounding; Adapt.: Adaptive-Blend.
Family
Clean step success
ASR
Clean reference
0.858
0.071
Patch
0.868
0.999
Popup
0.867
1.000
TypeText
0.864
0.988
Grounding
0.863
0.994
Adaptive-Blend
0.856
0.981
Table 18: Victim checkpoint quality. Clean step success and ASR on 1000 held-out steps each; all five meet the pre-specified criteria (drop ≤2 pt, ASR ≥0.80 ). Clean reference: chance rate.
TPR@1% FPR on followed steps / held-out clean FPR
Backbone
nfit/d
ASR
Clean acc.
IDEA (gated ρpc )
Log-ratio (ablation)
Earlier IDEA score (superseded)
Mahalanobis screen
Best other baseline
Qwen2.5-VL-3B
998/2048
0.999
0.868
1.000/0.017
1.000/0.011
0.015/0.006
1.000/0.011
OC-SVM 1.000/0.014
Qwen2.5-VL-7B
998/3584
1.000
0.858
0.999/0.020
0.999/0.013
0.109/0.005
1.000/0.014
OC-SVM 0.994/0.009
UI-TARS-1.5-7B
998/3584
1.000
0.890
0.921/0.023
0.996/0.026
0.221/0.006
1.000/0.013
OC-SVM 0.999/0.012
OS-Atlas-Base-7B
998/3584
0.998
0.812
1.000/0.012
1.000/0.013
0.078/0.014
0.999/0.009
OC-SVM 0.991/0.010
Table 19: Backbone sensitivity on Patch (overlapping single split, nfit=998 , 1% target). IDEA: frozen rule; “earlier IDEA score”: superseded whitened-space variant. OS-Atlas is Qwen2-VL-based, the others Qwen2.5-VL.
Patch
Ground.
Whole step, timed online
Agent alone
0.647
0.646
IDEA, clean step, early exit
0.672
0.677
IDEA, clean step, audit forced
1.403
1.398
IDEA, triggered step, early exit
1.414
1.435
Detector only, timed online
Table 20: Cost per step (median seconds, one GPU, Qwen2.5-VL-3B). Whole-step rows include the agent; detector-only rows do not. Protocol in Sec. R .
Method
Cal.
Step
Agnostic
Action
Loc.
FPR guar.
STRIP
✓
✓
✓
×
×
×
TeCo
✓
✓
✓
×
×
×
SCALE-UP
✓
✓
✓
×
×
×
IBD-PSC
✓
✓
✓
×
×
×
DeDe
✓
✓
✓
×
×
×
Mahalanobis
✓
✓
×
✓
×
×
Table 21: Requirements of the threat model, by the stated scope of each method. Cal.: clean-only calibration; Step: per-step decision; Agnostic: attack-agnostic; Action: GUI action output; Loc.: returns a region; FPR guar.: finite-sample FPR bound as published.
Detector
Adaptation
Strict split ( Table 1 ); all but the last five also on the earlier single split ( Table 8 )
TeCo [ 25 ]
Re-runs the agent under five corruption types (Gaussian noise, Gaussian blur, JPEG, brightness, contrast) at five severities; score = standard deviation over types of the first severity that changes the action
IBD-PSC [ 15 ]
Normalization weights of the last k decoder layers scaled by 1.5 for k=1,…,5 ; score = mean probability of the original first action token
DeDe [ 16 ]
A decoder is trained to reconstruct the screenshot from the visual tokens; score = pixel error against the 8× downsampled screenshot
Mahalanobis screen [ 17 ]
Mahalanobis distance of h0 under the empirical covariance of F (the gate of IDEA, used alone)
Entropy
Entropy of the model’s distribution over candidate actions (the agent’s action and its actions under two instruction paraphrases)
Table 22: Baseline detectors and their adaptations. h0 : decision state of the step; F : fit split; LW: Ledoit–Wolf shrinkage.
Strict split, IDEA alarms, 20 seeds, mean
Overlapping, RCA (ours)
Overlapping, earlier variant
Family
Rule
Hit
IoU >0.5
mIoU
H/I/m at α=0.005
Hit
IoU >0.5
mIoU
Hit
IoU >0.5
mIoU
Patch
point
0.862
0.673
0.546
0.863/0.673/0.547
0.859
0.670
0.544
0.295
0.205
0.193
Popup
bbox
0.993
0.000
0.160
0.993/0.000/0.160
0.994
0.000
0.160
0.536
0.000
0.082
Grounding
point
0.910
0.138
0.318
0.910/0.147/0.323
0.826
0.121
0.289
0.589
0.068
0.201
Grounding †
bbox
0.914
0.137
0.320
0.909/0.146/0.323
not run
not run
Adaptive-Blend
point
1.000
0.011
0.016
1.000/0.016/0.021
1.000
0.013
0.017
not run
Table 23: Localization. Strict split: correct alarms of the frozen rule at α=0.01 (and 0.005 ), mean over 20 seeds; overlapping split: one-seed earlier run. Hit: center of R⋆ inside the trigger box. † Scored on the log-ratio’s alarms.
Grounding
Adaptive-Blend
TypeText
@0.5%
@0.1%
@0.5%
@0.1%
@0.5%
@0.1%
Mahalanobis
0.630
0.090
0.903
0.744
1.000
1.000
Log-ratio ρ
0.456
0.256
0.063
0.026
0.922
0.718
Table 24: TPR at stricter operating points. Ungated Mahalanobis screen and ungated log-ratio ρ (ablation statistic, not the frozen rule) on the overlapping split at 0.5% and 0.1% target FPR.
Figure 7: IDEA needs about 400 clean calibration steps. TPR on followed steps (episode-disjoint split; mean and sd band over 20 seeds) with n calibration steps; “All” is the whole pool. Shaded: only the attainable level ( 1/26 , 1/51 ) is certified.
Family
n
IDEA
Log-ratio (abl.)
Screen
Grounding
50
0.812/0.027
0.819/0.027
0.769/0.038
100
0.869/0.012
0.882/0.014
0.501/0.011
200
0.891/0.008
0.905/0.007
0.514/0.009
400
0.952/0.008
0.929/0.008
0.542/0.010
All
0.962/0.008
0.950/0.008
0.532/0.009
Adaptive-Blend
50
0.773/0.023
0.667/0.020
0.869/0.032
Table 25: Calibration-size sweep on the episode-disjoint split. TPR on followed steps / pooled FPR T , mean over 20 seeds, at the effective level max(0.01,1/(nthr+1)) .
Example
n0
s0/T1
Emax
Sint
u
v
ρpc/Tα
IoU
(a) Patch
293.5
150.6 / 50.6
155.1
94.9
4.40
+ 5.54
7.07 / 0.64
0.54
(b) Popup
202.7
98.4 / 52.2
101.3
44.5
4.10
+ 9.46
10.32 / 0.27
0.16
(c) Grounding
103.1
53.6 / 51.6
47.7
61.4 ∗
1.35
+ 3.63
3.87 / 1.15
0.29
(d) TypeText
292.3
148.4 / 50.6
145.6
59.0 ∗
3.88
+ 8.61
9.45 / 0.50
0.40
(e) Adaptive-Blend
142.9
77.4 / 51.4
90.5
141.8
5.00
− 1.73
5.00 / 0.84
0.00
Table 26: Readout of the five alarms in Figure 5 . Gate distance s0 against T1 , the two scales, their residuals to the clean law, and the score against Tα . ∗ Fallback median.
The rapid adoption of mobile graphical user interface (GUI) agents, which autonomously control applications and operating systems (OS), exposes new system-level attack surfaces. Existing backdoors against web GUI agents and general GenAI models rely on environmental injection or deceptive pop-ups to mislead the agent operation. However, these techniques do not work on screenshots-based mobile GUI agents due to the challenges of restricted trigger design spaces, OS background interference, and conflicts in multiple trigger-action mappings. We propose AgentRAE, a novel backdoor attack capable of inducing Remote Action Execution in mobile GUI agents using visually natural triggers (e.g., benign app icons in notifications). To address the underfitting caused by natural triggers and achieve accurate multi-target action redirection, we design a novel two-stage pipeline that first enhances the agent's sensitivity to subtle iconographic differences via contrastive learning, and then associates each trigger with a specific mobile GUI agent action through a backdoor post-training. Our extensive evaluation reveals that the proposed backdoor preserves clean performance with an attack success rate of over 90% across ten mobile operations. Furthermore, it is hard to visibly detect the benign-looking triggers and circumvents eight representative state-of-the-art defenses. These results expose an overlooked backdoor vector in mobile GUI agents, underscoring the need for defenses that scrutinize notification-conditioned behaviors and internal agent representations.
Yutao Luo, Haotian Zhu, Shuchao Pang +4
Nanjing University of Science and Technology, China · Macquarie University, Australia · Western Sydney University, Australia +2
Large language models are often fine-tuned, shared, or downloaded from third parties, so a deployed model may carry a hidden backdoor that behaves normally on benign inputs but switches to attacker-controlled behavior when a secret trigger appears. While backdoors can be audited before deployment, runtime monitoring remains important for models that are frequently updated. The challenge is that LLM serving is latency-sensitive: existing inference-time detectors either rely on assumptions about the trigger form, which can fail on stealthy attacks, or require extra model computation, such as input perturbations or an additional generation pass. We introduce SpecGuard, an inference-time backdoor detector that repurposes speculative decoding at zero added model-computation cost. Speculative decoding speeds up inference by using a small draft model to propose tokens and a target model to verify them. We observe that this verification process already exposes a useful signal: when a backdoor is triggered, the target model shifts toward the attacker's behavior, while a clean draft model does not predict this shift, causing the draft-token acceptance rate to change. We formalize when this signal appears and show that an attacker who suppresses it must also weaken the backdoor. Across diverse backdoor types and model families, SpecGuard reliably detects triggered behavior, including stealthy cases where input-level filters are blind, while avoiding the extra generation cost of existing runtime detectors. Speculative decoding therefore doubles as a free, always-on signal for detecting backdoored LLM behavior.
Rui Wen, Ahmed Salem, Andrew Paverd +2
Institute of Science Tokyo · Microsoft Security Response Center · Microsoft Azure +1
Modern vision-language-model (VLM) based graphical user interface (GUI) agents are expected not only to execute actions accurately but also to respond to user instructions with low latency. While existing research on GUI-agent security mainly focuses on manipulating action correctness, the security risks related to response efficiency remain largely unexplored. In this paper, we introduce SlowBA, a novel backdoor attack that targets the responsiveness of VLM-based GUI agents. The key idea is to manipulate response latency by inducing excessively long reasoning chains under specific trigger patterns. To achieve this, we propose a two-stage reward-level backdoor injection (RBI) strategy that first aligns the long-response format and then learns trigger-aware activation through reinforcement learning. In addition, we design realistic pop-up windows as triggers that naturally appear in GUI environments, improving the stealthiness of the attack. Extensive experiments across multiple datasets and baselines demonstrate that SlowBA can significantly increase response length and latency while largely preserving task accuracy. The attack remains effective even with a small poisoning ratio and under several defense settings. These findings reveal a previously overlooked security vulnerability in GUI agents and highlight the need for defenses that consider both action correctness and response efficiency. Code can be found in https://github.com/tu-tuing/SlowBA.
Junxian Li, Tu Lan, Haozhen Tan +2
School of Computer Science, Shanghai Jiao Tong University