SPECTRA-VLA: Base-Fine Hyperspectral Token Transfer at Frozen Vision-Language-Action Interfaces

Chih-Chung Hsu

cs.CVOct 14, 2026

SPECTRA-VLA: Base-Fine Hyperspectral Token Transfer at Frozen Vision-Language-Action Interfaces

Authors: Chih-Chung Hsu

Organizations: National Yang Ming Chiao Tung University, Taiwan

Abstract

Hyperspectral imaging distinguishes RGB-similar objects, but a pretrained vision-language-action model needs a state its frozen decoder can use. SPECTRA-VLA provides a native-width spectral upgrade without retraining the visual encoder or action decoder. A learned continuous residual enriches each target token through its own normalized false-color reference; projection, norm control and validation-calibrated injection regulate the change. Across 86 held-out acquisitions, category balanced accuracy rises from 0.812 to 0.883. With identical residuals and scale, own-token addition improves reuse of an external category-derived probe on frozen decoder logits: 0.701 versus 0.590 with pooled replacement, at similar image-disjoint recognition. Five grouped tests covering the same 86 acquisitions recover within-category spectral shape with $R^2=0.802/0.337$ from continuous and posterior-code states. Category-matched spectral swaps further show that own-token composition approximately doubles the target-specific recovery increment after frozen decoding. Spectral enrichment and receiving-token preservation thus work together at the pretrained interface.

Figures & tables

Fig. 1: SPECTRA-VLA adds spectral residuals after the frozen visual backbone and pretrained projector. LN defines native-width reference

G

. Lower panels share

D,q,m_{t}

and skip

(1-m_{t})G_{t}

. For

q>0

, pooled

T_{q}=\mathrm{LN}_{\epsilon}(B^{\circ}+qD)

; only addition retains

G_{t}

inside the normalized branch. Pooled-token probes measure category and spectral shape. The frozen decoder consumes the full grid and generic prompt

p

; external first-position readouts measure four-state information and spectral recovery. Category labels train the mapper; validation selects

q

. Projection and norm control constrain the residual. Non-target tokens are unchanged;

q=0

exactly returns

G=\mathrm{LN}(Z_{\mathrm{base}})

. Blue/green identifies reference/spectral paths.

Method	Category bAcc	$\Delta$ Category [95% CI]	Token-proxy bAcc	$\Delta$ Token [95% CI]	Cos.
False-color base	0.812	$-0.071$ [ $-0.107,-0.050$ ]	0.860	$-0.056$ [ $-0.097,-0.033$ ]	1.000
High-anchor direct	0.814	$-0.069$ [ $-0.107,-0.047$ ]	0.862	$-0.055$ [ $-0.089,-0.032$ ]	0.991
SPECTRA-VLA (ours)	0.883	$0.000$	0.916	$0.000$	0.939
No orthogonalization	0.893	$+0.010$ [ $-0.002,0.023$ ]	0.929	$+0.013$ [ $0.002,0.035$ ]	0.937

TABLE I: SPECTRA-VLA improves recognition across 86 held-out acquisitions. Leave-one-session-out evaluation uses fold-local 12/60-epoch selection/refitting and one prediction per held-out object. Category/pooled-token proxy bAcc read pooled states; proxy labels are category-derived. Cos. measures false-color-reference similarity (all

\uparrow

). Differences are row minus SPECTRA-VLA, with conditional paired session 95% intervals. High-anchor direct requires validation cosine

\geq 0.94

. Gray identifies SPECTRA-VLA; bold marks favorable means, not significance.

Fig. 2: Injection scale and correspondence affect recognition and decoder-readout reuse differently. (a,b) Epoch-60, image-disjoint three-seed curves use refitted category/pair and reference-fixed decoder probes. Table III changes checkpoint/scale, retaining the reference-trained probe. (c–e) Correct-minus-control differences retain correct-spectrum-trained readouts (“spectrum-fixed” in e) and

q=(2,4,4)

for seeds 0–2. Both (b,e) use recomputed decoder outputs; correct-spectrum bAcc in (e) is

0.710

. Filled/open markers denote intervals excluding/including zero. Session 95% intervals are simultaneous across five controls per endpoint; pairs use 38-session-node product multipliers.

Image split: 4096-D tokens, 461 objects
Representation	bAcc $\uparrow$	Within-category $R^{2}\uparrow$	Angle $\downarrow$
False-color base	–	0.263	0.041
Continuous (ours)	–	0.824	0.019
Category posterior	–	0.374	0.038
Hard category code	–	0.333	0.039
Continuous–posterior $\Delta R^{2}=0.450\ [0.398,0.504]$

TABLE II: Continuous tokens retain more object-specific spectral variation than category codes. All blocks use validation-CE checkpoints and allowance

0.2

. Recovery

R^{2}

and angle (radians) use train-fitted category conditioning (Eq. ( 12 )); category bAcc uses the unconditioned category readout (Section 5.2). Held-session results pool five groups over 86 acquisitions; the final mapper comparison uses 16 sessions. Paired intervals resample acquisitions. Learned rows average three seeds; bAcc is shown where jointly evaluated. Bold favors means within blocks; gray identifies SPECTRA-VLA.

Composition	Category $\uparrow$	Reference-fixed $\uparrow$	Refit $\uparrow$	Disp. P95 $\downarrow$	KL P95 $\downarrow$
Base-fine (ours)	0.897	0.701	0.791	0.513	1.158
Pooled replacement	0.892	0.590	0.741	0.362	5.593

TABLE III: Own-token composition improves readout reuse and target-specific recovery. Arms share residuals/scales and recomputed logits. (a,b) Image-disjoint: 461 objects, CE-selected epochs

(3,6,4)

q=4

, reference bAcc

0.695

. (c,d) Five held-out groups: 2,291 objects/86 sessions, reference

0.681

, unrestricted validation-utility-selected scales. (d) Each cell separately train-fits centering, standardization and ridge; increment is own-minus-same-category-donor within-category

R^{2}

. Equal seed contributions share acquisition resampling, including donor nodes in (d); secondary intervals are unadjusted. Disp.: pooled movement; P95: per seed. Gray: base-fine; bold: favorable means, not significance. Supplement S2.4 gives influence.

Method	Cat. bAcc $\uparrow$	Disp. P95 $\downarrow$	KL P95 $\downarrow$
Base-fine	0.880	0.188	0.152
No pooled-base projection	0.880	0.213	0.182

TABLE IV: Recognition and disturbance at validation-CE checkpoints, using 461 objects from 66 sessions and the same category ridge grid. (a) Allowance

0.2

gives achieved validation means

0.1438/0.1620

. Matching achieved mean target

0.15

instead gives base-fine/no-projection category bAcc

0.871/0.871

and KL P95

0.163/0.160

(Supplement S3.3). (b) Both inputs use the same composition and allowance

0.2

. Disp. and decoder KL are per-seed test 95th percentiles;

R^{2}

reads within-category spectral shape. Gray identifies continuous SPECTRA-VLA. Bold marks favorable rounded means, not significance.

Fig. 3: Category repair, injection-induced harm, unresolved recognition and target-region failure on fixed-test examples. True RGB marks targets. Lower views (a–c) enlarge targets for display only; side views show original full-frame inputs at unchanged intensities. Refit category predictions are bold when correct. In (d), ground-truth (solid gold) and predicted (dashed cyan) regions have IoU $0.0029$ ; the ground-truth-region-trained decoder-logit probe changes from correct to wrong. Spectra (a–c) use each target’s peak; both (d) curves share the larger region peak. Wavelengths approximate an uncalibrated uniform grid. Outcome-selected cases do not estimate prevalence.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Category (ID)	State	Train/Val./Test
Film (1)	1	203/67/66
Basket (2)	2	200/54/67
Cardboard (3)	0	35/15/10
Video tape (4)	3	397/117/130
Filament (5)	1	57/20/16
Bag (6)	1	500/169/172

Appendix

TABLE S1: Waste-object categories, object counts and the fixed four-state proxy mapping. Category IDs and names follow SpectralWaste; the grouping is defined by the evaluation.

Condition	Category bAcc	Pair	C-proxy bAcc	Correct-spectrum-fixed bAcc	Cosine
Correct target fine	0.905	0.778	0.938	0.711	0.939
Zero fine (fixed readout)	0.727	0.481	0.699	0.635	1.000
No fine loss (fixed)	0.868	0.691	0.889	0.681	0.939
Wrong-target fine	0.459	0.185	0.514	0.587	0.939
Sample-shuffled fine	0.510	0.235	0.521	0.586	0.939
Class-mismatched fine	0.337	0.074	0.449	0.587	0.939

Appendix

TABLE S2: Correct spectral correspondence supports recognition and decoder information. (a) Correct-spectrum-fixed readouts at epoch 60. (b) Simultaneous paired intervals over five cached-logit controls: shared image/session clusters, both endpoint-session memberships for pairs. (c) Joint recomputation with unchanged correct-spectrum heads; session intervals are primary, image intervals sensitivity. SD measures optimization spread.

Input	Mean $q$	Cat.	Proxy	Cos.
Continuous spectrum	4.00	0.897	0.931	0.919
Hard category code	1.08	0.893	0.896	0.993
Six-class posterior	2.00	0.901	0.916	0.969

Appendix

TABLE S3: Continuous spectra and matched category codes offer distinct operating points. (a) Utility-selected scales with refitted pooled readouts. (b) Common validation displacement allowance

0.2

, with jointly recomputed fixed/refitted decoder probes. The code classifier adds 26,182 parameters; paired decoder uncertainty appears in Table S8 .

Input	Setting	$R^{2}\uparrow$	Seed SD	RMSE $\downarrow$	Angle $\downarrow$
False-color base	4096-D, allowance 0.2	0.263	0.000	0.906	0.041
Continuous spectra (ours)	4096-D, allowance 0.2	0.824	0.023	0.442	0.019
Category posterior	4096-D, allowance 0.2	0.374	0.007	0.835	0.038
Hard category code	4096-D, allowance 0.2	0.333	0.006	0.862	0.039
False-color base	256-D, allowance 0.2	0.217	0.000	0.933	0.043
Continuous spectra (ours)	256-D, allowance 0.2	0.609	0.065	0.659	0.030

Appendix

TABLE S4: Continuous payloads retain spectral shape beyond category identity. (a) Fixed-state recovery and uniformly expanded ridge-grid sensitivity. RMSE uses standardized targets; angle is in radians. (b) Paired cluster intervals; the first session comparison is primary within this post-hoc analysis. (c) Recovery from the 256 frozen decoder logits at the same allowance-

0.2

states. Seed SD is optimization spread.

State	$\boldsymbol{R^{2}\uparrow}$	Angle $\downarrow$	Category bAcc $\uparrow$
Continuous spectrum	0.802	0.020	0.861
Category posterior	0.337	0.038	0.882
False-color base	0.238	0.040	0.793

Appendix

TABLE S5: Spectral content and composition across five held-out groups/86 sessions. (a) Allowance-

0.2

recovery/category bAcc; angle is in radians. (b) Shared residuals; fold-local scales maximize unrestricted validation utility. Reference-fixed probes retain reference training; each refit probe trains on its cell’s logits, including donors. Donors change correspondence, preserving category. Values pool out-of-fold objects after equal seed averaging.

Paired difference	$\boldsymbol{\Delta}$	Acquisition 95% CI
Spectral content $\times$ composition	+0.0171	$[0.0121,0.0228]$
Addition: own $-$ donor spectral recovery	+0.0336	$[0.0268,0.0411]$
Replacement: own $-$ donor spectral recovery	+0.0165	$[0.0123,0.0210]$
Reference-fixed: addition $-$ replacement	+0.0257	$[0.0004,0.0572]$
Refit: addition $-$ replacement	+0.0350	$[0.0148,0.0656]$
Reference-fixed: injection $\times$ composition	+0.0088	$[-0.0070,0.0362]$

Appendix

TABLE S6: Paired uncertainty and class contributions across 86 acquisitions. (a) Shared resampling, 10,000 draws, seed 20261012: node-product multipliers for donor contrasts, multinomial session bootstrap otherwise. Seeds contribute equally. Content-by-composition and reference-fixed addition/replacement are primary; other intervals are unadjusted. Spectral rows: within-category

R^{2}

; state rows: bAcc. (b) Refitted proxy-state recall/support. Bold marks higher displayed recall, including ties; contributions differ from sampling uncertainty.

Family	$\boldsymbol{\alpha/q}$	Cosine	Category bAcc	Pair	C-proxy bAcc	Reference-fixed bAcc
Direct	0.000	0.181	0.899	0.716	0.925	0.458
Direct	0.010	0.285	0.898	0.728	0.929	0.462
Direct	0.050	0.604	0.898	0.716	0.930	0.550
Direct	0.100	0.736	0.901	0.691	0.926	0.586
Direct	0.250	0.791	0.903	0.691	0.930	0.600
Direct	0.500	0.808	0.906	0.679	0.927	0.618

Appendix

TABLE S7: Complete fixed-epoch curves. (a) Retargeting varies alignment

\alpha

; base-fine varies injection

q

, with cached reference-fixed logit probes. (b) Validation-high-anchor base-conditioned replacement versus selected base-fine. (c) Common-

\alpha

curves, distinct from per-seed selection. Higher recognition/cosine and lower KL are favorable.

Fig. S1: Epoch-60 fixed-split anchoring–retention sensitivity. (a) Spectrum-only retargeting and base-fine on 27 stress pairs. (b) The shared reference-fixed decoder-logit probe on 461 objects, using cached logits with reference bAcc

0.704

. Curves are descriptive three-seed means over observed ranges; pair response is nonmonotone. Selected base-fine gives pair resolution

0.778

at reference cosine

0.939

Seed 0; validation-selected scale 4
Composition	Category $\uparrow$	Proxy $\uparrow$	Fixed $\uparrow$	Cached-trained fixed $\uparrow$	Refit $\uparrow$	KL P95 $\downarrow$
Base-fine	0.884	0.928	0.701	0.707	0.757	1.096
Pooled replacement	0.883	0.927	0.603	0.606	0.725	5.598
Seed 1; validation-selected scale 4
Base-fine	0.900	0.933	0.690	0.687	0.788	1.176
Pooled replacement	0.896	0.918	0.574	0.576	0.734	5.584

Appendix

TABLE S8: Receiving-token composition matters at closely matched recognition. (a) Three optimization seeds with identical residuals/checkpoints/scales; KL P95 uses the jointly recomputed reference. (b) Paired acquisition-session intervals on 461 objects in 66 sessions; all contrasts are exploratory. Positive bAcc and negative KL differences favor the first condition.

Spectral input	Composition	Category $\uparrow$	Proxy $\uparrow$	Fixed $\uparrow$	Refit $\uparrow$
None	Original token grid	0.801	0.857	0.695	0.695
None	Pooled replacement	0.787	0.857	0.597	0.694
Continuous	Token-wise base-fine	0.897	0.931	0.701	0.791
Continuous	Pooled replacement	0.892	0.924	0.590	0.741

Appendix

TABLE S9: Composition and supervision controls. (a) Same-residual four-cell comparison with and without injection in the jointly recomputed runtime. Fixed preserves the reference-trained decoder readout. (b) Complete final-epoch matched ablation with refitted cached-logit probes; Pair is both-endpoint correctness, Category/Proxy state are pooled bAcc and Cosine is reference anchoring. Zero scale is a reference endpoint.

Spectral input	Active dim.	Pair	Category	Proxy	Cos.
Raw bands	120	0.7531	0.9163	0.9372	0.9584
Train-only NMF	16	0.7531	0.9077	0.9123	0.9486
Raw+NMF	136	0.7531	0.9187	0.9426	0.9285

Appendix

TABLE S10: Input and execution calibration. (a) Capacity-matched raw/NMF inputs, with active width reported separately; Pair 27 uses the central stress pairs. (b) Validation-CE checkpoint states. (c) Validation mean-displacement allowances; reported tails are measured on test. Decoder values in (b,c) use cached-logit sensitivity, separate from joint recomputation.

Method	Category	Proxy
No projection	0.891 [0.864, 0.928]	0.929 [0.909, 0.940]
Operational base	0.789 [0.769, 0.803]	0.826 [0.803, 0.857]
Base-fine	0.891 [0.874, 0.927]	0.929 [0.909, 0.940]

Appendix

TABLE S11: Readout robustness after fixing the representations. Mean [minimum, maximum] bAcc spans seven predeclared transformations and three training seeds; ranges are not sampling confidence intervals.

Variant	$q$	Norm ratio $D$ / $qD$	Cos.	KL
$\rho=0.1$ (main)	3.33	0.100/0.333	0.939	1.400
No orthogonalization	3.00	0.100/0.300	0.951	0.250
No norm bound	2.00	0.671/1.333	0.655	7.148
$\rho=0.025$	4.00	0.025/0.100	0.994	0.085
$\rho=0.05$	4.00	0.050/0.200	0.977	0.427
$\rho=0.2$	3.67	0.200/0.733	0.800	4.828

Appendix

TABLE S12: Constraint sensitivity. (a) Fixed-split final-epoch budget/component ablation with refitted cached-logit probes. Ratios distinguish pre-scale

\|D\|/\|B\|

and injected

q\|D\|/\|B\|

; displayed

q

averages validation-selected scales. (b) Separate 5/15-epoch nested constraint study; Sel. counts inner-validation choices, and the decoder probe is reference-fixed.

Fig. S2: Image-disjoint constraint sensitivity at the final epoch, using cached logits and refitted decoder probes (Table S12 a). Bars compare

\rho=0.1

, no orthogonalization and no norm bound. The budget sweep separates recognition, anchoring and accessible decoder information; selected scale partly compensates for budget. Error bars are SD across three seeds.

Fig. S3: Nested constraint diagnostic: five selection and 15 refit epochs. (a) Inner-validation choices. (b,c) Recognition–anchoring and reference-fixed probe–movement, with session-clustered 95% intervals. (d) Overlapping low-KL settings separated on rows; estimates and intervals are unchanged. Selected constraints retain high anchoring without large-budget probe degradation.

Method	Cat.	Proxy	Cos.	F1	Session mean/median/worst-10
False-color base	0.812	0.860	1.000	0.760	0.792/0.826/0.401
High-anchor direct	0.814	0.862	0.991	0.796	0.806/0.839/0.433
SPECTRA-VLA (ours)	0.883	0.916	0.939	0.863	0.893 / 0.936 / 0.612
No orthogonalization	0.893	0.929	0.937	0.877	0.892/0.933/0.584

Appendix

TABLE S13: Recognition gains across unseen acquisitions. (a) Principal 12/60-epoch comparison on the same 2,291 out-of-fold objects; category macro-F1 and equal-weight session summaries expand the main results. Session means/medians average present-class recalls; Worst-10 averages each method’s ten lowest sessions. (b) Category support and recall. Gray identifies SPECTRA-VLA; bold marks favorable means, not significance.

Method	Cosine 0.75 ( $n=27$ )	Cosine 0.80 ( $n=9$ )
Base	0.593 [0.388, 0.776]	0.556 [0.212, 0.863]
Raw spectrum	0.630 [0.424, 0.806]	0.556 [0.212, 0.863]
Direct retargeter	0.691 [0.460, 0.862]	0.593 [0.212, 0.925]
Base-fine	0.778 [0.537, 0.937]	0.741 [0.299, 0.972]
Base-fine (no pair weighting)	0.765 [0.498, 0.937]	0.667 [0.299, 0.925]

Appendix

TABLE S14: Ambiguity-definition sensitivity. (a) Threshold-specific inventories; resolution averages seed estimates, and exact-CI envelopes span per-seed intervals rather than providing pooled coverage. (b) Continuous similarity over 215 fixed object-disjoint pairs, with differences averaged across seeds before inference. Its session-node intervals use 20,000 draws (seed 20260805).

Fig. S4: Threshold sensitivity with separately reconstructed inventories:

0.75

yields 27 pairs,

0.80

yields 9, and stricter thresholds yield none. Points are three-seed means; whiskers span per-seed exact pair-binomial intervals and are descriptive rather than a pooled 95% interval.

Metric	$\Delta$ [95% CI]	Seed SD
Fixed-split base contrast
Pair	+0.185 [-0.030,+0.443]	0.037
Category bAcc	+0.104 [+0.058,+0.154]	0.013
Hard bAcc	+0.059 [-0.013,+0.141]	0.032
C-proxy bAcc	+0.081 [+0.022,+0.147]	0.005
Fixed-split direct contrast (exploratory)

Appendix

TABLE S15: Paired differences (S6). Fixed-split decoder/prototype rows: epoch-60 representation-refitted readouts, pooled scores. Nested rows: reference-fixed readouts, equal-weight session means (not pooled OOF). Accuracy up/KL down favors base-fine; bold excludes zero. Base/direct contrasts are primary/exploratory. Direct-pair CIs are descriptive; other CIs use indicated image/session units. Seed SD: optimization spread.

Representation	Refitted probe Acc.	Refitted probe bAcc	Ref.-fixed bAcc / symbol agreement	Category lookup bAcc	256-KL
Majority	0.551	0.250	–	–	–
Zero visual	0.282	0.250	–	0.250	–
True RGB token	0.751	0.656	–	0.615	5.251
False-color base	0.781	0.704	0.704 / 1.000	0.726	0.000
Raw+NMF fixed projection ( $136\!\to\!4096$ )	0.850	0.800	–	0.775	5.042
Spectral fine only	0.803	0.799	–	0.790	9.397

Appendix

TABLE S16: Fixed-epoch decoder readouts using cached logits. (a) Refitted and category-lookup readouts, plus one reference-trained ridge with regularization 10 applied unchanged. The combined column reports its four-state bAcc, then native action-symbol argmax agreement with the false-color reference over 256 logits. (b) Proxy recall after category prediction and lookup. Means use three seeds; higher bAcc/agreement and lower reference-to-altered KL are favorable.

Method	Fixed bAcc	F1	Session mean/median/worst-10	W/L/T
False-color base	0.705	0.640	0.682 / 0.750 / 0.162	–
Direct retargeter	0.560	0.533	0.621/0.651/0.081	27/43/16
Concatenation-projection	0.447	0.362	0.391/0.442/0.000	8/72/6
SPECTRA-VLA (ours)	0.684	0.612	0.658/0.725/0.142	18/38/30

Appendix

TABLE S17: Short-budget session-held-out reference-fixed decoder diagnostic. Each representation is inner-selected, refitted for 15 epochs and tested once per held-out object. Pooled OOF bAcc uses 2,291 predictions; equal-weight session summaries average present-class recalls. This five/15-epoch study differs from principal 12/60 recognition and the fixed-rank constraint experiment.

Readout fitting	Raw	$G$	Injected
Raw-trained, fixed	0.730	0.647	0.611
$G$ -trained, fixed	0.609	0.704	0.620
State-specific refit	0.730	0.704	0.711

Appendix

TABLE S18: Decoder-coordinate decomposition and paired uncertainty. (a) Fixed versus state-specific readouts on identical objects/coordinates. (b) Conditional paired intervals for OpenVLA coordinate changes and SpatialVLA gains; 10,000 shared 66-session draws (seed 20260922), with absent-class recall zero. Bold marks intervals excluding zero, including adverse effects.

Complement seed	KL (action/complement ratio)	Difference [95% session CI]
20260813	0.086515 (6.029 $\times$ )	0.435093 [0.385371, 0.495584]
20260814	0.086559 (6.026 $\times$ )	0.435049 [0.385856, 0.497977]
20260815	0.083752 (6.228 $\times$ )	0.437856 [0.387819, 0.498721]

Appendix

TABLE S19: Separate normalized-input vocabulary diagnostic (Section S6.3). Action-coordinate movement versus three preset vocabulary complements at

q=4

. Each 256-coordinate subset is internally normalized from the same full-vocabulary forward. Objectwise means across adapter seeds precede paired acquisition-session inference. Complement sets are descriptive controls rather than a vocabulary-wide null.

Quartile	Disagree (%)	$T_{q}$ / $C_{q}$ margin	$\Delta$
1	2.84	5.890/5.593	-0.297
2	3.51	4.910/4.857	-0.053
3	1.93	5.586/5.833	+0.247
4	3.21	6.157/6.818	+0.661

Appendix

TABLE S20: Auxiliary-head behavior by validation occupancy quartile (validation-CE epochs

3,6,4

;

q=4

). Margins are true-category logits minus the largest competitor, reported as

T_{q}/C_{q}

. Section S7.2 separately gives epoch-60 geometry.

SpatialVLA representation	Cat.	Pair	C-proxy	Cos.	Base-fixed FP	Full KL	Top- 1 agr.
False-color base	0.812	0.704	0.871	1.000	0.718	0.000	1.000
Direct retargeter	0.836	0.642	0.888	0.229	0.588	4.529	0.155
Base-fine	0.914	0.827	0.947	0.985	0.673	0.709	0.522
No orthogonalization	0.894	0.753	0.928	0.974	0.703	0.594	0.611

Appendix

TABLE S21: SpatialVLA replication. (a) Matched states with refitted pooled heads and one reference-fixed decoder probe. KL/top-1 use all 8,194 action coordinates. (b) Correspondence controls retain correct-spectrum pooled heads; the action head remains reference-fixed. At zero scale the grid/logits equal the base, though pooled scores use different heads.

GT (IoU=1.000)
Representation	Acc.	bAcc	F1
Base	0.781	0.704	0.642
Base-fine	0.759	0.711	0.628
Erode (IoU=0.763)
Base	0.759	0.683	0.617
Base-fine	0.772	0.732	0.649

Appendix

TABLE S22: Grounding sensitivity with fixed ground-truth-region-trained decoder readouts and scales. Every condition re-encodes its region. IoU compares with the target mask; predicted masks receive the target category and select its largest predicted component.

Fig. S5: Region re-encoding on 461 objects in the earlier cached decoder configuration (Section S1.6). Each representation’s ground-truth-mask-trained probe and scale remain fixed across masks. Probes use 256 first-position coordinates; points are descriptive means without sampling intervals. Erosion preserves base-fine/base bAcc

0.732/0.683

; largest predicted components supplied with the target category ( † ) give IoU

0.218

and bAcc

0.362/0.468

SPECTRA-VLA (raw-only)
Rank	Parameters	Selected scale	Category bAcc	Pair	C-proxy bAcc	Cosine
1	53621	1.583	0.866	0.667	0.884	0.993
2	57838	2.000	0.872	0.722	0.898	0.993
4	66272	4.000	0.890	0.815	0.908	0.981
8	83140	2.333	0.881	0.741	0.909	0.991
16	116876	2.667	0.888	0.833	0.902	0.990

Appendix

TABLE S23: Capacity and spectral-input sensitivity. (a) Raw-only rank sweep, three seeds, validation-selected scales; counts exclude frozen VLA/readouts. (b) Session-held-out band exclusion with unchanged false-color tokens; only the fine-path input changes. Higher category/proxy/pair/cosine is favorable.

Test rendering	Category bAcc	Pair
Low/mid/high	0.929	0.827
NIR edge	0.918	0.852
Random 3-band	0.929	0.840
Fixed sensor	0.917	0.815
Gain/offset	0.913	0.840
PCA	0.918	0.827

Appendix

TABLE S24: Six-render augmentation under one fixed scale/readout per seed, with three-seed category/pair means. All test views use the same 461 objects and 27 pairs; this is separate from unaugmented primary evaluation.

Fig. S6: Rendering and radiometric robustness after six-view training. Fixed sensor denotes the primary reference bands

[119,60,0]

. Category bAcc spans

0.913

–

0.929

and pair resolution

0.815

–

0.852

. Points are seeds 0–2; squares/whiskers are means and seed SD, not sampling confidence intervals.

Base-fine
Seed	Retained	Repaired	Acc.
0	0.986	0.469	0.879
1	0.973	0.583	0.892
2	0.967	0.573	0.885
Mean	0.975 [0.961, 0.987]	0.542 [0.437, 0.639]	0.885
Base-conditioned direct

Appendix

TABLE S25: Conditional recognition relative to the false-color base. Seed rows give aligned 461-object proportions; mean retention/correction includes acquisition-session intervals.

P(B+)

denotes base correctness.

Class	False-color base	Direct retargeter	SPECTRA-VLA
Film	0.758	0.929	0.894
Basket	0.776	0.905	0.876
Cardboard	0.800	1.000	1.000
Video tape	0.792	0.846	0.887
Filament	0.875	0.875	0.896
Bag	0.802	0.835	0.876

Appendix

TABLE S26: Class-resolved fixed-split recall, with methods compared in columns. Categories and proxy states follow Table S1 . Decoder-logit rows use refitted probes on frozen logits.

	1	2	3	4	5	6
1	0.758	0.000	0.000	0.000	0.030	0.212
2	0.045	0.776	0.045	0.075	0.000	0.060
3	0.100	0.000	0.800	0.100	0.000	0.000
4	0.023	0.069	0.008	0.792	0.092	0.015
5	0.000	0.000	0.000	0.125	0.875	0.000
6	0.128	0.029	0.017	0.006	0.017	0.802

Appendix

TABLE S27: Row-normalized confusions (three-seed means), with methods side by side. Rows: true labels; columns: predictions. (a) Category IDs 1–6 follow Table S1 . (b,c) Its proxy states 0–3 are read from pooled states and refitted decoder-logit probes, respectively. Bold highlights diagonals, not cross-method best values; unrounded diagonals reproduce recall and bAcc.

Fully loaded pretrained encoders
Feature (Dim.)	Acc.	bAcc	F1	Hard	Pair
HyperSL (256)	0.770	0.787	0.721	0.803	0.667
MaskedSST (96)	0.768	0.767	0.692	0.818	0.519
SpectralGPT (768)	0.866	0.868	0.824	0.818	0.593
Raw spectral reference
Raw spectrum (120)	0.766	0.795	0.741	0.775	0.630

Appendix

TABLE S28: Spectral separability and dataset-specific material recognition. (a) Published frozen features use the common training-only readout; Hard is hard-subset bAcc. Partial-load diagnostics ( † DOFA, ‡ HyperSIGMA) are not ranked; ‡ also denotes out-of-domain pretraining. (b) Wood uses ten development splits and one class-balanced official test, so bAcc equals Acc.; SpecTex uses one within-source split without pairs. Fine width

k

is validation-selected. † HyperSIGMA is partially loaded. Dim. is feature width and F1 is macro-F1.

Fig. S7: Dataset-specific material augmentation. Width

k

indexes the fine feature. Wood dots/squares/whiskers are runs/means/seed-split SD over ten development splits. SpecTex uses one within-source split across all 60 arrays. These are expanded material-readout features rather than native-width VLA states.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models
Aug 3, 2026Jin Cui, Chuanchang Su, Jiayi Lu +3Visual Question Answering VLM Robustness
1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Computer Science and Technology, Xi’an Jiaotong University · School of Software Engineering, Xi’an Jiaotong University
WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models
Aug 9, 2026Zikang Wen, Yuning Zhang, Dong YuanEfficient VLA Model Inference Speculative Decoding
The University of Sydney
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
Jul 12, 2026Zhaoyang Li, Yanjun Li, Wangkai Li +2Efficient VLM Inference Visual Token Pruning
School of Information Science and Technology / National Key Laboratory of Deep Space Exploration, University of Science and Technology of China.

Aug 3, 2026J/K move · Enter open · S save

HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

Jin Cui, Chuanchang Su, Jiayi Lu, Xinyue Long+2

1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Computer Science and Technology, Xi’an Jiaotong University · School of Software Engineering, Xi’an Jiaotong University

Visual Question Answering VLM Robustness Vision-Language Models VLM Adaptation

Open paper AskPDF arXiv

Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response rigidity. Despite substantial frequency variation across images and tasks, pretrained vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM attention to effectively utilize the enriched visual tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.

Abstract

Vision-Language-Action Models

Explore similar work

HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models

Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models