cs.LGOct 18, 2026

Finite Interaction Lens: Structured Precision Sharing for Vision-Language Models

Authors: Nozomu Fujisawa

Organizations: Keio University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Identifying which visual interventions interact in a vision-language model requires comparing their joint and separate effects. The resulting four-output difference can be small enough for rounding error to obscure it. We introduce the Finite Interaction Lens (FIL), a mixed-precision method for evaluating finite intervention families. At each selected linear projection, FIL decomposes each candidate's input into a native state, two single-intervention components, and a full-dimensional mixed residual. It projects the single-intervention components at higher precision and reuses the same numerical contributions in the joint candidates, while retaining every mixed residual and nonlinear trajectory. We derive a local cancellation identity and a conditional bound on error propagation through a smooth nonlinearity. In one MLP of Qwen3-VL-4B-Instruct, FIL reduces final interaction MSE by 94.4% on 16 synthetic families and by 99.85% and 99.35% in two separate four-image COCO studies, relative to a control that has the same nominal BF16-product budget and higher-precision single interventions but projects each joint candidate independently. One image accounts for most of the absolute gain in each COCO study. Higher-accuracy multi-product BF16 arithmetic on all candidates gives lower aggregate error at similar total query times. Single-product BF16 already recovers the reference choices in both candidate-selection tasks we tested. On these development inputs, FIL improves interaction fidelity at a fixed nominal precision budget. Its practical value depends on a task that needs the extra fidelity.

Figures & tables

Figure 1: FIL within decoder 13. (a) The gate/up projections use structured arithmetic; each candidate retains its nonlinear suffix and KV history. (b) Native, axis, and full mixed projections reconstruct all 64 candidates. The same stored axis projections enter the axis and joint outputs.
Figure 2: Local projection-error structure. Under exact component formation and reconstruction (Proposition 1), shared native and axis errors cancel in the mixed difference. The bottom row shows the additional finite-precision terms from Eq. ( 24 ).
MethodMean interaction MSETime vs. history dense (ms)
History dense1.77×10−121.77\times 10^{-12}0.0
Delta BF1 ( 14+4914+49 )7.97×10−107.97\times 10^{-10}−4.0-4.0
Axes-only BF35.02×10−105.02\times 10^{-10}−2.5-2.5
Factored BF19.72×10−109.72\times 10^{-10}−1.9-1.9
FIL (factored axes BF3)2.81×10−112.81\times 10^{-11}−1.0-1.0
Delta BF33.53×10−123.53\times 10^{-12}−2.3-2.3
Table 1: Synthetic single-MLP evaluation: mean interaction MSE over 16 families, and mean of seven-repeat medians over four timed families relative to history dense (1671.2 ms). Delta BF1 ( 14+4914+49 ) and delta BF1 (63 packed) are two packings of the same computation; the BF3 rows use the optimized implementations.
Figure 3: Synthetic evaluation. (a) Mean MSE over 16 families versus mean query time over four family medians. Time is relative to history dense (1671.2 ms); negative values indicate lower total query time. (b) Paired family losses on equal logarithmic scales. The dashed diagonal marks parity.
MethodMean interaction MSETime vs. history dense (ms)
History dense6.96×10−126.96\times 10^{-12}0.0
Delta BF18.34×10−98.34\times 10^{-9}−7.0-7.0
Axes-only BF37.72×10−97.72\times 10^{-9}−6.1-6.1
FIL1.16×10−111.16\times 10^{-11}−5.3-5.3
Delta BF37.71×10−127.71\times 10^{-12}−4.9-4.9
Plain BF31.08×10−111.08\times 10^{-11}−5.4-5.4
Table 2: Natural-image evaluation: four image means, each averaging two questions. Time averages seven-repeat case medians within image, then across images, and is given relative to history dense (1405.4 ms). The BF3-based methods use batched audit read-back; Table 14 lists both read-back variants. Image 61 is not included.
ImageAxes-only MSEFIL MSEReduction (%)Share (%)Omit: reduction (%)
08.11×10−118.11\times 10^{-11}3.50×10−123.50\times 10^{-12}95.690.25299.86
164.90×10−164.90\times 10^{-16}2.87×10−162.87\times 10^{-16}41.566.61×10−76.61\times 10^{-7}99.85
324.94×10−114.94\times 10^{-11}7.20×10−137.20\times 10^{-13}98.540.15899.85
483.08×10−83.08\times 10^{-8}4.23×10−114.23\times 10^{-11}99.8699.5996.77
Table 3: Image-level interaction MSE, FIL versus axes-only BF3 with batched auditing. Share is the fraction of the total absolute MSE reduction; Omit is the relative reduction over the other three images.
Figure 4: Natural-image evaluation with batched auditing. (a) MSE for each image, averaged over two questions. (b) Mean error and query time relative to history dense (1405.4 ms); negative times indicate lower total cost. Original-audit variants are listed in Table 14 .
MethodInteraction MSEMax probability errorTime vs. history dense (ms)
History dense2.99×10−122.99\times 10^{-12}2.26×10−52.26\times 10^{-5}0.0
Delta BF15.37×10−95.37\times 10^{-9}1.24×10−31.24\times 10^{-3}−14.9-14.9
Axes-only BF33.97×10−93.97\times 10^{-9}1.24×10−31.24\times 10^{-3}−7.3-7.3
FIL2.57×10−112.57\times 10^{-11}1.72×10−41.72\times 10^{-4}−3.5-3.5
Delta BF33.40×10−123.40\times 10^{-12}2.40×10−52.40\times 10^{-5}−2.0-2.0
Plain BF32.89×10−122.89\times 10^{-12}2.39×10−52.39\times 10^{-5}−6.0-6.0
Table 4: Region-pair study: four images, six pairs per image, one MLP. Approximate methods use the common per-image precision range. MSE is an image mean of pair means. Time is relative to history dense (7749.9 ms), using the mean of seven-repeat image medians for six-pair bundles. Time differences are calculated before rounding. Max error is over all states and five groups.
IndexCOCO IDAxes-only MSEFIL MSEPlain BF3 MSEShare (%)
05298501.50×10−81.50\times 10^{-8}9.92×10−119.92\times 10^{-11}6.20×10−126.20\times 10^{-12}94.53
33027897.48×10−107.48\times 10^{-10}2.87×10−122.87\times 10^{-12}4.20×10−124.20\times 10^{-12}4.72
133155771.20×10−101.20\times 10^{-10}7.42×10−137.42\times 10^{-13}1.17×10−121.17\times 10^{-12}0.75
222219116.37×10−166.37\times 10^{-16}1.87×10−171.87\times 10^{-17}2.18×10−172.18\times 10^{-17}3.92×10−63.92\times 10^{-6}
Table 5: Region-pair study image means, each over six pairs. These indices refer to a different inventory from the two-question study in Table 21 ; COCO IDs identify the images. Share is the fraction of total axes-only-minus-FIL MSE reduction.
Figure 5: Region-pair study at a common precision range. (a) Image-mean interaction MSE over six pairs, labeled by COCO ID. (b) Each dot is an image’s seven-repeat median time minus its same-run history-dense median; diamonds are means of four paired differences. Mean history-dense bundle time is 7.75 s.
MethodDiagonal sumSigned cross sumInteraction MSE
Axes-only BF35.09×10−105.09\times 10^{-10}−7.43×10−12-7.43\times 10^{-12}5.02×10−105.02\times 10^{-10}
FIL2.89×10−112.89\times 10^{-11}−8.03×10−13-8.03\times 10^{-13}2.81×10−112.81\times 10^{-11}
Table 6: Final-output uncentered error moments, averaged over 16 synthetic families. The cross sum includes the contrast signs from Eq. ( 28 ).
Figure 6: Error propagation on 16 synthetic families. (a) Each point is a family and each diamond is the ratio of mean losses, both against FP32 references; the dashed line marks parity. (b) Axes-only minus FIL in the diagonal, signed cross, and total terms of Eq. ( 28 ).
VariantMean interaction MSEMSE / FILTime −- FIL (ms)
FIL, cached8.03×10−118.03\times 10^{-11}1.0000.0
FIL, same-shape axis recomputation8.03×10−118.03\times 10^{-11}1.000+5.7+5.7
BF3 on fixed mixed 142.66×10−92.66\times 10^{-9}33.17+1.4+1.4
BF3 on norm-selected mixed 142.91×10−92.91\times 10^{-9}36.23+2.5+2.5
Factored all BF31.33×10−121.33\times 10^{-12}0.0165+3.5+3.5
FIL at gate only, up dense2.93×10−112.93\times 10^{-11}0.365+5.3+5.3
Table 7: Ablations on four synthetic families, relative to FIL measured in the same run (MSE 8.03×10−118.03\times 10^{-11} ). The two mixed-14 controls keep the nominal BF3 budget; the final three rows change it or the set of approximated projections.
Figure 7: Precision and caching ablations on four families. Both panels use the same-run FIL baseline in Table 7 . Rows below the divider change the precision or projection budget. Time differences average within-family seven-repeat medians.
MethodMean interaction MSETime vs. history dense (ms)
History dense1.50×10−121.50\times 10^{-12}0.0
Delta BF19.69×10−109.69\times 10^{-10}−7.2-7.2
Axes-only BF3, original audit8.34×10−108.34\times 10^{-10}−5.1-5.1
FIL, original audit8.03×10−118.03\times 10^{-11}−5.0-5.0
Delta BF3, original audit1.48×10−121.48\times 10^{-12}−3.2-3.2
Plain BF3, original audit1.93×10−121.93\times 10^{-12}−4.2-4.2
Table 8: Audit comparison on four synthetic families: mean interaction MSE and mean of within-family seven-repeat median query times, relative to history dense (1636.3 ms). Both audit variants are measured in the same run.
Figure 8: Audit comparison on four synthetic families. (a) Same-run means from Table 8 , with time relative to history dense (1636.3 ms). Open and filled markers distinguish original and batched auditing; one label identifies each method. Dotted segments connect observed nondominated points. (b) Batched minus original audit time, measured separately for the local MLP and whole query.
MethodComparisons with g>2εg>2\varepsilonObserved top-1 agreement
Delta BF175/8080/80
FIL79/8080/80
Plain BF379/8080/80
Table 9: Post hoc selection margins on the 16 synthetic families and five output groups, relative to the FP32 reference, for three methods of the storage-reuse run (Appendix E ). All 13 methods of that run, including the ten not listed, select correctly in all 80 comparisons.
Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

MethodMean interaction MSEPrimary query time (ms)
History dense1.77×10−121.77\times 10^{-12}1634.862
Delta BF1 packed7.97×10−107.97\times 10^{-10}1627.608
Delta BF1 storage7.97×10−107.97\times 10^{-10}1627.512
Axes-only BF3 packed5.02×10−105.02\times 10^{-10}1630.131
Axes-only BF3 storage5.02×10−105.02\times 10^{-10}1629.602
Factored BF1 packed9.72×10−109.72\times 10^{-10}1631.247
Appendix
Table 10: Packing/storage comparison: 16-family accuracy and four-family timing. Times average seven-repeat family medians, including source cost. All methods use the audited zero-product optimization.
StatisticPackedStorageStorage −- packed
Mean of four family medians1631.7971631.364−0.433-0.433
Mean of all 28 repetitions1631.7091635.087+3.378+3.378
Appendix
Table 11: FIL timing sensitivity in the same storage-reuse run. No repetition is excluded. All entries are in milliseconds.
Figure 9: Storage-reuse comparison. (a) Selected methods from Table 10 : 16-family accuracy and four-family timing, relative to history dense (1634.862 ms). (b) Storage minus packed time: per-family differences of medians, their mean, and the difference of the arithmetic means of all 28 repetitions, which has the opposite sign.
Condition or contrastFILDelta BF3
No packing, separate preparation: p0s0p_{0}s_{0}24.8323.45
No packing, shared preparation: p0s1p_{0}s_{1}22.1720.80
Packing, separate preparation: p1s0p_{1}s_{0}24.1423.14
Packing, shared preparation: p1s1p_{1}s_{1}21.4920.50
Table 1 projection path, local timing23.7922.35
Packing with shared preparation: p1s1−p0s1p_{1}s_{1}-p_{0}s_{1}−0.682-0.682−0.298-0.298
Appendix
Table 12: Local-MLP packing/preparation 2×22\times 2 ablation. Cell means and paired differences average four families, in milliseconds. The difference of differences is (p1s1−p0s1)−(p1s0−p0s0)(p_{1}s_{1}-p_{0}s_{1})-(p_{1}s_{0}-p_{0}s_{0}) . Contrasts are computed before rounding.
MethodSynchronizations, original →\to batchedLocal MLP Δ\DeltaWhole-query Δ\Delta
FIL24→1224\to 12−0.542-0.542+0.080+0.080
Axes-only BF324→1224\to 12−0.366-0.366−0.322-0.322
Delta BF315→815\to 8−0.215-0.215−0.888-0.888
Plain BF315→815\to 8−0.214-0.214+0.113+0.113
Appendix
Table 13: Batched minus original audit retrieval within the audit-batching comparison. Synchronization counts are mean profiler counts over four families. Local and whole-query time changes are in milliseconds, measured separately.
MethodMean interaction MSEPrimary query time (ms)
History dense6.96×10−126.96\times 10^{-12}1405.438
Delta BF18.34×10−98.34\times 10^{-9}1398.411
Axes-only BF3, original audit7.72×10−97.72\times 10^{-9}1399.489
FIL, original audit1.16×10−111.16\times 10^{-11}1400.179
Delta BF3, original audit7.71×10−127.71\times 10^{-12}1400.362
Plain BF3, original audit1.08×10−111.08\times 10^{-11}1400.378
Appendix
Table 14: Natural-image run with both audit read-back variants: four image means, each averaging two questions, and the mean over images of seven-repeat case medians. Matched original and batched variants give identical logits.
MethodInteraction MSEMax errorTime −- history dense (ms)
History dense7.29×10−127.29\times 10^{-12}3.16×10−53.16\times 10^{-5}0.000
Plain BF12.06×10−72.06\times 10^{-7}8.41×10−38.41\times 10^{-3}−32.843-32.843
Delta BF12.53×10−82.53\times 10^{-8}1.32×10−21.32\times 10^{-2} †−31.324-31.324
Plain BF39.42×10−129.42\times 10^{-12}6.98×10−56.98\times 10^{-5}−14.656-14.656
Delta BF36.88×10−126.88\times 10^{-12}3.62×10−53.62\times 10^{-5}−12.912-12.912
Split8 BF12.56×10−112.56\times 10^{-11}9.21×10−59.21\times 10^{-5}−11.245-11.245
Appendix
Table 15: Four-MLP natural-image baseline evaluation: 128 cases in 64 image clusters, four MLPs. The two questions per image receive equal weight. Interaction MSE is the image-mean five-group loss; Max error is the global maximum probability error over cases, states, and groups; † exceeds the 0.01 gate. Time differences use within-case seven-repeat medians on eight images with two questions each, then average by image. FIL is not evaluated here.
Figure 10: Four-MLP natural-image baselines, without FIL. (a) Image-mean paired MSE over 64 images. (b) Maximum error for all 128 cases. The star marks the parent image of case 123, which exceeds the fixed 0.01 gate.
Figure 11: Signed five-group probability errors for case 123 at i=5,j=1i=5,j=1 , with the fixed ±0.01\pm 0.01 gate. The OTHER error exceeds the gate. The case remains in all 128-case aggregates.
IndexImage IDABIndexImage IDAB
02740831402241686523230221911659062211551
1137004153889915584073341213678954718171
23467521531577191381834382207904801537396
3241326911347558357065710414142109880
45247751385711680613637546110460441536142
54801411478137103371979153817182048
Appendix
Table 16: Complete COCO val2014 inventory for the four-MLP baseline study. A and B are annotation IDs. Indices are local to this inventory; the region-pair study uses the different images in Table 22 .
MethodPairLoss ↓\downarrowInteraction MSE ↓\downarrowMax error ↓\downarrowTotal msΔ\Delta ms vs. history dense
History dense3.73 ⁣× ⁣10−123.73\!\times\!10^{-12}3.04 ⁣× ⁣10−123.04\!\times\!10^{-12}3.032 ⁣× ⁣10−53.032\!\times\!10^{-5}1406.097—
Plain BF14.17 ⁣× ⁣10−84.17\!\times\!10^{-8}3.32 ⁣× ⁣10−83.32\!\times\!10^{-8}5.296 ⁣× ⁣10−35.296\!\times\!10^{-3}1384.171−21.926-21.926
Delta BF14.88 ⁣× ⁣10−94.88\!\times\!10^{-9}3.71 ⁣× ⁣10−93.71\!\times\!10^{-9}2.030 ⁣× ⁣10−32.030\!\times\!10^{-3}1385.366−20.731-20.731
Plain BF34.24 ⁣× ⁣10−124.24\!\times\!10^{-12}3.74 ⁣× ⁣10−123.74\!\times\!10^{-12}3.131 ⁣× ⁣10−53.131\!\times\!10^{-5}1400.289−5.808-5.808
Delta BF33.41 ⁣× ⁣10−123.41\!\times\!10^{-12}3.27 ⁣× ⁣10−123.27\!\times\!10^{-12}3.105 ⁣× ⁣10−53.105\!\times\!10^{-5}1401.352−4.745-4.745
Split8 BF11.96 ⁣× ⁣10−111.96\!\times\!10^{-11}2.26 ⁣× ⁣10−112.26\!\times\!10^{-11}8.596 ⁣× ⁣10−58.596\!\times\!10^{-5}1403.517−2.581-2.581
Appendix
Table 17: Native-relative baselines: 64 synthetic families, four MLPs. Losses are family means; Max error is the global five-group maximum. Time averages seven-repeat medians for eight families.
Figure 12: Post hoc tolerance curve for the 64-family native-relative study. Both BF3 variants pass all 64 families at every threshold shown; the original gate is 0.01.
RankGateUp
327.09%7.09\%3.70%3.70\%
6410.61%10.61\%6.61%6.61\%
12816.14%16.14\%11.71%11.71\%
Appendix
Table 18: Static low-rank correction: median error-energy recovery on the opposite calibration fold (16 development families).
MethodInteraction RMSEndpoint RMS
Delta BF14.54 ⁣× ⁣10−64.54\!\times\!10^{-6}2.97 ⁣× ⁣10−62.97\!\times\!10^{-6}
Factored BF11.48 ⁣× ⁣10−71.48\!\times\!10^{-7}2.99 ⁣× ⁣10−62.99\!\times\!10^{-6}
Axes-only BF33.17 ⁣× ⁣10−63.17\!\times\!10^{-6}2.77 ⁣× ⁣10−62.77\!\times\!10^{-6}
Factored axes BF35.13 ⁣× ⁣10−85.13\!\times\!10^{-8}3.91 ⁣× ⁣10−83.91\!\times\!10^{-8}
Delta BF33.23 ⁣× ⁣10−83.23\!\times\!10^{-8}1.68 ⁣× ⁣10−81.68\!\times\!10^{-8}
Dense FP323.26 ⁣× ⁣10−83.26\!\times\!10^{-8}1.62 ⁣× ⁣10−81.62\!\times\!10^{-8}
Appendix
Table 19: CPU probe at t=1/16t=1/16 : FP32 execution against dense FP64. RMS pools 32 families in one generated network.
Figure 13: CPU amplitude sweep against dense FP64. Shared-axis methods show approximately second-order decay in the fixed FP64 fit window; FP32 curves reach a numerical floor. The network has two RMSNorm–SwiGLU residual blocks and no attention.
CallsArraysShapesScales
1One-layer weight(64,32)(64,32)1/641/\sqrt{64}
2–4One-layer x,a,bx,a,b(512,64)(512,64) each11
5–7Layer 1 gate, up, down(32,48),(32,48),(48,32)(32,48),(32,48),(48,32)1/32,1/32,0.3/481/\sqrt{32},1/\sqrt{32},0.3/\sqrt{48}
8–10Layer 2 gate, up, down(32,48),(32,48),(48,32)(32,48),(32,48),(48,32)1/32,1/32,0.3/481/\sqrt{32},1/\sqrt{32},0.3/\sqrt{48}
11Output head(32,4)(32,4)0.4/320.4/\sqrt{32}
12–14Two-layer x0,a,bx_{0},a,b(32,1,1,32)(32,1,1,32) each11
Appendix
Table 20: Complete random-draw order for the CPU probe. Rows containing several calls are executed left to right. No random draws occur between these calls. Scales are applied after drawing.
IndexCOCO image IDA: category / annotationB: category / annotationRole
0274083car / 140224bus / 168652Main
1664307cup / 676954bowl / 710873Main
32302219backpack / 1165906handbag / 2211551Main
4859652dog / 6832potted plant / 23168Main
61108669bowl / 1535648apple / 1548458Recheck
Appendix
Table 21: Natural inputs from COCO val2014. Category/annotation pairs identify donor A and B masks and questions q=0,1q=0,1 . Image 61 is used only to recheck the failure case of Appendix G .
IndexCOCO IDRegions 0–3
0529850cat (50214); person (231125); suitcase (1182980); refrigerator (1646710)
3302789backpack (1167322); suitcase (1185989); bed (1611695); refrigerator (1646862)
13315577dog (11796); couch (111776); laptop (1099186); book (1147889)
22221911orange (1051333); cup (674996); cake (1088518); knife (1892941)
Appendix
Table 22: Region-pair study stimuli from COCO val2014. Regions are listed in order 0–3, as category (annotation ID); region 0 is queried. These indices are local to this study.
IndexCOCO IDIdentical groups / 29Max input difference
05298501/291.43×10−61.43\times 10^{-6}
33027898/291.19×10−61.19\times 10^{-6}
133155778/291.79×10−61.79\times 10^{-6}
222219118/291.10×10−61.10\times 10^{-6}
Appendix
Table 23: Cross-pair input audit at the common precision range in the region-pair study. Max difference is across repeated copies of the same native or single-region state after normalization.
ColorToken ID: decoded string
red1151: red ; 2518: ␣red ; 3731: ␣Red ; 6033: Red ; 6504: RED
green6176: ␣green ; 7840: ␣Green ; 13250: green ; 19576: Green ; 55954: GREEN
blue6303: ␣blue ; 8697: ␣Blue ; 10331: Blue ; 12203: blue ; 64330: BLUE
yellow13753: ␣yellow ; 25462: ␣Yellow ; 27869: yellow ; 47699: Yellow ; 97029: YELLOW
Appendix
Table 24: Fixed color-token groups. Each entry gives token ID followed by its decoded string. The symbol ␣ denotes exactly one leading ASCII space; all other characters are literal.
Index iiLayoutNative classesDonor ADonor BFrontierActive
2083BBGYB →\to GC →\to B40178
2091GGBBD →\to GA →\to B40178
2101GGBGD →\to RA →\to B40178
2113YRYGA →\to RB →\to G12494
2121GBBBC →\to YD →\to Y12494
2130GRRYA →\to BB →\to G40178
Appendix
Table 25: All 16 synthetic configurations. Scene ID is 133000000+i133000000+i and seed is 164191009+i164191009+i . Native class tuples are ordered A–D; R/G/B/Y denote red/green/blue/yellow. “Donor A” gives the queried/first edited label and its new class; “Donor B” gives the separately edited label and its new class. Frontiers are zero-based; active count is 218−218- frontier.
IndexNative ANative BNative CNative DDonor A replacement
208(59,79,202)(41,80,193)(62,177,81)(204,186,28)(28,187,64)
209(30,188,53)(63,195,78)(50,62,197)(54,65,225)(46,192,65)
210(38,194,61)(30,191,56)(41,72,214)(52,188,71)(193,57,48)
211(223,193,35)(187,33,43)(224,204,56)(40,178,76)(211,50,61)
212(54,182,75)(57,66,203)(50,60,204)(30,91,202)(218,213,27)
213(63,175,80)(202,60,28)(194,34,51)(207,195,38)(57,75,223)
Appendix
Table 26: Recorded RGB fills, ordered A–D, and replacement RGB for donor A. Donor B uses the exact palette RGB of the class listed in Table 25 . Values are unsigned image-channel integers.
Figure 14: Synthetic family 208: native and separate donor images, drawn from the recorded layout and RGB values. Typography is schematic. Donor A changes panel B, donor B changes panel C, and the question asks about B. The intervention grid combines the supported hidden differences.
More in this topic

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 404 papers

Explore similar work

CardsList
Aug 1, 2026

Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations

Vision-language models (VLMs) process image patches and text tokens in a shared residual stream, but the local geometry through which the two modalities interact remains poorly understood. Most interpretability methods identify global linear directions, which may miss representations that are globally high-dimensional but locally low-dimensional. We introduce LENS (Local Explanation of Neighborhood Subspaces), a method that decomposes VLM activations into local low-rank Gaussian neighborhoods using a Mixture of Factor Analyzers. Applied to LLaVA-1.5-7B and Qwen3-VL-8B, LENS reveals distinct depth-dependent fusion trajectories consistent with each model's fusion mechanism: LLaVA progressively mixes modalities at later layers, whereas Qwen3-VL mixes them early, partially re-segregates them, and recombines them near the output. An automated multimodal labeling pipeline assigns concise semantic descriptions to these neighborhoods. Interpolating activations toward neighborhood centroids causally redirects generation within and across modalities and outperforms difference-in-means and VL-SAE in most evaluated conditions; in one LLaVA vision-to-vision setting, MFA achieves 5.7 times the VL-SAE score. Human evaluation finds MFA steering competitive with prompting and substantially stronger than the other intervention baselines. Finally, the MFA coefficient space improves Qwen3-VL image-to-rendered-text retrieval at the deepest evaluated layer from 14.9% to 48.6% R@1. Ablations show that the reported fusion trajectories are stable across component counts, local ranks, and modality-purity thresholds. These results support local geometric neighborhoods as useful interpretable and causal units for analyzing cross-modal representations in the evaluated VLMs.

Shalom Kachko, Raz Lapid, Margarita Vald +2

1Ben-Gurion University of the Negev · 2Intuit Inc. · 3Deepkeep

Vision-Language ModelsRepresentation Geometry in VLMsVLM InterpretabilityVision-Language AlignmentActivation Steering
AskPDF
Sep 25, 2026

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

We present FLIP, a final-layer inference-time probe for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation. Behavioral change under internal intervention is otherwise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hidden state before logit computation, leaving parameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negligible change, a bounded interior regime in which detection recall at IoU 0.50 (R50R_{50}) improves while tolerant counting error (Ecount\mathcal{E}_{\mathrm{count}}) falls, and over-suppression. We formalize a four-criterion probe-and-sweep protocol for disciplining the interpretation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions, including the last-block output before final normalization, and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multiple VLMs replicate it. FLIP is therefore a validation step for intervention-based mechanistic interpretability, not a steering method. Repo at https://github.com/earl-juanico/flip-qwen3vl/

Drandreb Earl O. Juanico, Rowel O. Atienza

AI Graduate Program, University of the Philippines, Diliman, Quezon City, Philippines 1101 · EEE Institute, College of Engineering, University of the Philippines, Diliman, Quezon City, Philippines 1101

VLM InterpretabilityVLM Evaluation
AskPDF
Jun 12, 2026

Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Models

Fine-tuning vision-language models to emit dense coordinate lists improves visual grounding but also changes how models serialize, repeat, and terminate structured outputs. We study this behavior as a generation and control surface. In Gemma 4 12B, high-capacity q/k/v/o LoRA raises class-aware [email protected] from 0.007 to 0.448 while inducing repeated-tail pressure (duplicate rate 0.080, max repeat 23). A q/v rank sweep keeps max repeat at 21-22 across ranks 4-64, showing capacity persistence. The target signal is separable: object-level repeat-stop removes exact repeated records (duplicate rate 0.000, max repeat 1) while preserving F1 (0.494 to 0.490) and stricter [email protected] (0.381 to 0.385). Structure-axis probes localize the effect to bbox-coordinate object lists; dense non-bbox and spatial/count JSON remain repeat-clean, including under high-capacity adapters. Qwen3-VL-8B reproduces a clean controlled endpoint ([email protected] 0.318, duplicate rate 0.000), and COCO 2017 reproduces acquisition plus duplicate pressure. Dense coordinate-list adaptation therefore creates a structure-bound, cross-family interference surface that can be measured and controlled.

Chenyu Zhou, Qiliang Jiang, Boguang Pan

School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Graduate School of Information, Production and Systems, Waseda University

VLM AdaptationVision-Language GroundingStructured Output Generation
AskPDF