Finite Interaction Lens: Structured Precision Sharing for Vision-Language Models
Organizations: Keio University
Abstract
Identifying which visual interventions interact in a vision-language model requires comparing their joint and separate effects. The resulting four-output difference can be small enough for rounding error to obscure it. We introduce the Finite Interaction Lens (FIL), a mixed-precision method for evaluating finite intervention families. At each selected linear projection, FIL decomposes each candidate's input into a native state, two single-intervention components, and a full-dimensional mixed residual. It projects the single-intervention components at higher precision and reuses the same numerical contributions in the joint candidates, while retaining every mixed residual and nonlinear trajectory. We derive a local cancellation identity and a conditional bound on error propagation through a smooth nonlinearity. In one MLP of Qwen3-VL-4B-Instruct, FIL reduces final interaction MSE by 94.4% on 16 synthetic families and by 99.85% and 99.35% in two separate four-image COCO studies, relative to a control that has the same nominal BF16-product budget and higher-precision single interventions but projects each joint candidate independently. One image accounts for most of the absolute gain in each COCO study. Higher-accuracy multi-product BF16 arithmetic on all candidates gives lower aggregate error at similar total query times. Single-product BF16 already recovers the reference choices in both candidate-selection tasks we tested. On these development inputs, FIL improves interaction fidelity at a fixed nominal precision budget. Its practical value depends on a task that needs the extra fidelity.
Figures & tables
| Method | Mean interaction MSE | Time vs. history dense (ms) |
|---|---|---|
| History dense | 0.0 | |
| Delta BF1 ( ) | ||
| Axes-only BF3 | ||
| Factored BF1 | ||
| FIL (factored axes BF3) | ||
| Delta BF3 |
| Method | Mean interaction MSE | Time vs. history dense (ms) |
|---|---|---|
| History dense | 0.0 | |
| Delta BF1 | ||
| Axes-only BF3 | ||
| FIL | ||
| Delta BF3 | ||
| Plain BF3 |
| Image | Axes-only MSE | FIL MSE | Reduction (%) | Share (%) | Omit: reduction (%) |
|---|---|---|---|---|---|
| 0 | 95.69 | 0.252 | 99.86 | ||
| 16 | 41.56 | 99.85 | |||
| 32 | 98.54 | 0.158 | 99.85 | ||
| 48 | 99.86 | 99.59 | 96.77 |
| Method | Interaction MSE | Max probability error | Time vs. history dense (ms) |
|---|---|---|---|
| History dense | 0.0 | ||
| Delta BF1 | |||
| Axes-only BF3 | |||
| FIL | |||
| Delta BF3 | |||
| Plain BF3 |
| Index | COCO ID | Axes-only MSE | FIL MSE | Plain BF3 MSE | Share (%) |
|---|---|---|---|---|---|
| 0 | 529850 | 94.53 | |||
| 3 | 302789 | 4.72 | |||
| 13 | 315577 | 0.75 | |||
| 22 | 221911 |
| Method | Diagonal sum | Signed cross sum | Interaction MSE |
|---|---|---|---|
| Axes-only BF3 | |||
| FIL |
| Variant | Mean interaction MSE | MSE / FIL | Time FIL (ms) |
|---|---|---|---|
| FIL, cached | 1.000 | 0.0 | |
| FIL, same-shape axis recomputation | 1.000 | ||
| BF3 on fixed mixed 14 | 33.17 | ||
| BF3 on norm-selected mixed 14 | 36.23 | ||
| Factored all BF3 | 0.0165 | ||
| FIL at gate only, up dense | 0.365 |
| Method | Mean interaction MSE | Time vs. history dense (ms) |
|---|---|---|
| History dense | 0.0 | |
| Delta BF1 | ||
| Axes-only BF3, original audit | ||
| FIL, original audit | ||
| Delta BF3, original audit | ||
| Plain BF3, original audit |
| Method | Comparisons with | Observed top-1 agreement |
|---|---|---|
| Delta BF1 | 75/80 | 80/80 |
| FIL | 79/80 | 80/80 |
| Plain BF3 | 79/80 | 80/80 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Mean interaction MSE | Primary query time (ms) |
|---|---|---|
| History dense | 1634.862 | |
| Delta BF1 packed | 1627.608 | |
| Delta BF1 storage | 1627.512 | |
| Axes-only BF3 packed | 1630.131 | |
| Axes-only BF3 storage | 1629.602 | |
| Factored BF1 packed | 1631.247 |
| Statistic | Packed | Storage | Storage packed |
|---|---|---|---|
| Mean of four family medians | 1631.797 | 1631.364 | |
| Mean of all 28 repetitions | 1631.709 | 1635.087 |
| Condition or contrast | FIL | Delta BF3 |
| No packing, separate preparation: | 24.83 | 23.45 |
| No packing, shared preparation: | 22.17 | 20.80 |
| Packing, separate preparation: | 24.14 | 23.14 |
| Packing, shared preparation: | 21.49 | 20.50 |
| Table 1 projection path, local timing | 23.79 | 22.35 |
| Packing with shared preparation: |
| Method | Synchronizations, original batched | Local MLP | Whole-query |
|---|---|---|---|
| FIL | |||
| Axes-only BF3 | |||
| Delta BF3 | |||
| Plain BF3 |
| Method | Mean interaction MSE | Primary query time (ms) |
|---|---|---|
| History dense | 1405.438 | |
| Delta BF1 | 1398.411 | |
| Axes-only BF3, original audit | 1399.489 | |
| FIL, original audit | 1400.179 | |
| Delta BF3, original audit | 1400.362 | |
| Plain BF3, original audit | 1400.378 |
| Method | Interaction MSE | Max error | Time history dense (ms) |
|---|---|---|---|
| History dense | 0.000 | ||
| Plain BF1 | |||
| Delta BF1 | † | ||
| Plain BF3 | |||
| Delta BF3 | |||
| Split8 BF1 |
| Index | Image ID | A | B | Index | Image ID | A | B |
|---|---|---|---|---|---|---|---|
| 0 | 274083 | 140224 | 168652 | 32 | 302219 | 1165906 | 2211551 |
| 1 | 137004 | 1538899 | 1558407 | 33 | 412136 | 78954 | 718171 |
| 2 | 346752 | 1531577 | 1913818 | 34 | 382207 | 90480 | 1537396 |
| 3 | 241326 | 9113 | 47558 | 35 | 70657 | 1041414 | 2109880 |
| 4 | 524775 | 138571 | 168061 | 36 | 375461 | 1046044 | 1536142 |
| 5 | 48014 | 11478 | 137103 | 37 | 197915 | 38171 | 82048 |
| Method | PairLoss | Interaction MSE | Max error | Total ms | ms vs. history dense |
|---|---|---|---|---|---|
| History dense | 1406.097 | — | |||
| Plain BF1 | 1384.171 | ||||
| Delta BF1 | 1385.366 | ||||
| Plain BF3 | 1400.289 | ||||
| Delta BF3 | 1401.352 | ||||
| Split8 BF1 | 1403.517 |
| Rank | Gate | Up |
|---|---|---|
| 32 | ||
| 64 | ||
| 128 |
| Method | Interaction RMS | Endpoint RMS |
|---|---|---|
| Delta BF1 | ||
| Factored BF1 | ||
| Axes-only BF3 | ||
| Factored axes BF3 | ||
| Delta BF3 | ||
| Dense FP32 |
| Calls | Arrays | Shapes | Scales |
|---|---|---|---|
| 1 | One-layer weight | ||
| 2–4 | One-layer | each | |
| 5–7 | Layer 1 gate, up, down | ||
| 8–10 | Layer 2 gate, up, down | ||
| 11 | Output head | ||
| 12–14 | Two-layer | each |
| Index | COCO image ID | A: category / annotation | B: category / annotation | Role |
|---|---|---|---|---|
| 0 | 274083 | car / 140224 | bus / 168652 | Main |
| 16 | 64307 | cup / 676954 | bowl / 710873 | Main |
| 32 | 302219 | backpack / 1165906 | handbag / 2211551 | Main |
| 48 | 59652 | dog / 6832 | potted plant / 23168 | Main |
| 61 | 108669 | bowl / 1535648 | apple / 1548458 | Recheck |
| Index | COCO ID | Regions 0–3 |
|---|---|---|
| 0 | 529850 | cat (50214); person (231125); suitcase (1182980); refrigerator (1646710) |
| 3 | 302789 | backpack (1167322); suitcase (1185989); bed (1611695); refrigerator (1646862) |
| 13 | 315577 | dog (11796); couch (111776); laptop (1099186); book (1147889) |
| 22 | 221911 | orange (1051333); cup (674996); cake (1088518); knife (1892941) |
| Index | COCO ID | Identical groups / 29 | Max input difference |
|---|---|---|---|
| 0 | 529850 | 1/29 | |
| 3 | 302789 | 8/29 | |
| 13 | 315577 | 8/29 | |
| 22 | 221911 | 8/29 |
| Color | Token ID: decoded string |
|---|---|
| red | 1151: red ; 2518: ␣red ; 3731: ␣Red ; 6033: Red ; 6504: RED |
| green | 6176: ␣green ; 7840: ␣Green ; 13250: green ; 19576: Green ; 55954: GREEN |
| blue | 6303: ␣blue ; 8697: ␣Blue ; 10331: Blue ; 12203: blue ; 64330: BLUE |
| yellow | 13753: ␣yellow ; 25462: ␣Yellow ; 27869: yellow ; 47699: Yellow ; 97029: YELLOW |
| Index | Layout | Native classes | Donor A | Donor B | Frontier | Active |
|---|---|---|---|---|---|---|
| 208 | 3 | BBGY | B G | C B | 40 | 178 |
| 209 | 1 | GGBB | D G | A B | 40 | 178 |
| 210 | 1 | GGBG | D R | A B | 40 | 178 |
| 211 | 3 | YRYG | A R | B G | 124 | 94 |
| 212 | 1 | GBBB | C Y | D Y | 124 | 94 |
| 213 | 0 | GRRY | A B | B G | 40 | 178 |
| Index | Native A | Native B | Native C | Native D | Donor A replacement |
|---|---|---|---|---|---|
| 208 | (59,79,202) | (41,80,193) | (62,177,81) | (204,186,28) | (28,187,64) |
| 209 | (30,188,53) | (63,195,78) | (50,62,197) | (54,65,225) | (46,192,65) |
| 210 | (38,194,61) | (30,191,56) | (41,72,214) | (52,188,71) | (193,57,48) |
| 211 | (223,193,35) | (187,33,43) | (224,204,56) | (40,178,76) | (211,50,61) |
| 212 | (54,182,75) | (57,66,203) | (50,60,204) | (30,91,202) | (218,213,27) |
| 213 | (63,175,80) | (202,60,28) | (194,34,51) | (207,195,38) | (57,75,223) |
Large Vision-Language Models
Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.
See all 404 papers