AROMA: Adaptive Representation-Oriented Multimodal Actuation for Vision-Language Model Control
Organizations: University of Toronto
Abstract
Mechanistic localization does not by itself yield usable control: causal sensitivity may occupy little activation energy, an accurate local direction may fail at finite scale, and a reachable numeral target may be wrong. We introduce AROMA (Adaptive Representation-Oriented Multimodal Actuation), a frozen-VLM control framework separating where to intervene, which geometry and gain to use, how to reach a target, which target to pursue, and when to act. In Llama-3.2-11B-Vision-Instruct, four dimensions of a causally validated cross-attention head explain 89.82% of cardinality sensitivity but only 9.52% of median activation energy; magnitude-normalized causal subspace actuation converts this mismatch into finite control, improving exact-count accuracy on untouched procedural data by +8.00 pp. Iterative Minimal-Effort Actuation (MEA) reaches 81.25% of oracle targets versus 15.18% for a frozen direction under matched step budgets. With target and gate frozen before evaluation, targeted MEA improves the previous frozen policy from 34.444% to 38.730% on a new natural Stage-2 cohort (N=630; paired p=4.63e-7) and yields 65.741% versus 65.153% over all 7,992 examples while editing only 63 inputs. Prospective routing, replication across heads and models, and comparisons with attribution-steering baselines further characterize each stage, separating localization, actuation, reachability, target quality, and selective deployment as distinct control problems.
Figures & tables
Appendix figures & tables50 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Primary result | Interpretation | Evidence class | |
|---|---|---|---|---|
| Llama directional characterization | 300 | 91.33% down; 90.67% up; 88.67% strict | Primary bidirectional causal actuator | Held-out characterization |
| Llama Top-4 geometry | 300 / 653 | 89.82% sensitivity ( ); 9.52% median projected energy ( audit) | Sensitivity–energy mismatch | Mechanism validation |
| Llama untouched v4 | 2000 | 50.25% base; 58.25% MN-CSA; | Same-policy low-rank utility | Prospective untouched |
| E1 continuous local response | 160 | at : Pearson , Spearman , sign 100% | First-order projected control is locally accurate | Mechanism validation |
| Oracle MEA reachability | 112 | 81.25% target success; frozen direction 15.18%; activation-aligned 0.89% | Re-linearization converts local geometry into global reachability | Controller validation |
| Utility Router untouched final | 294 | 60.54% global MN 66.33%; | Low-rank candidate routing | Prospective untouched |
| Claim | Primary evidence | Status |
|---|---|---|
| Llama L18H13 is a bidirectional cardinality actuator | v1 intervention directionality, dose response, identity check, held-out candidate-vs-random validation | Mechanistic development + held-out validation |
| Bidirectional actuator behavior is not unique to L18H13 | frozen seven-head historical pool; fresh common protocol; 6/7 bidirectional replications, including 5/6 non-primary | Prospective multi-actuator robustness confirmation |
| Llama cardinality sensitivity is strongly low-rank | rank-4 total/centered spectra; 10,000 norm-matched null replicates | Mechanistic observation; conservative promotion criterion retained as unsatisfied |
| Naive Llama low-rank actuation under-actuates | same- failure plus label-free strength audit | Developmental diagnosis |
| Learned Llama Top-4 directions are behaviorally privileged | MN-CSA recovery, Bottom-4 control, 20 pre-frozen random rank-4 controls | Developmental specificity evidence |
| Frozen Llama MN-CSA improves unseen procedural counting | newly generated v4, zero seed/pixel overlap, frozen primary endpoint | Confirmatory |
| 0 | .25 | .5 | .75 | 1 | 1.25 | 1.5 | 2 | 3 | 4 | |
|---|---|---|---|---|---|---|---|---|---|---|
| Repairs | 20 | 16 | 13 | 8 | 0 | 18 | 35 | 60 | 80 | 89 |
| 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 | |
|---|---|---|---|---|---|---|---|---|
| Total second moment | 0.762162 | 0.841718 | 0.898229 | 0.944624 | 0.977195 | 0.993469 | 0.999231 | 1.000000 |
| Centered residual | 0.695283 | 0.797791 | 0.867775 | 0.927978 | 0.970304 | 0.991503 | 0.999001 | 1.000000 |
| Count | Base | Whole | MN | Whole–Base | MN–Base | MN–Whole |
|---|---|---|---|---|---|---|
| 1 | 1.000 | 1.000 | 1.000 | 0.0 | 0.0 | 0.0 |
| 2 | .995 | 1.000 | 1.000 | +0.5 | +0.5 | 0.0 |
| 3 | .880 | .960 | .940 | +8.0 | +6.0 | -2.0 |
| 4 | .750 | .860 | .850 | +11.0 | +10.0 | -1.0 |
| 5 | .515 | .690 | .730 | +17.5 | +21.5 | +4.0 |
| 6 | .425 | .545 | .565 | +12.0 | +14.0 | +2.0 |
| Slope | Pearson | Spearman | MAE | Sign agreement | |
|---|---|---|---|---|---|
| 0.05 | 1.0573 | 0.9936 | 0.9990 | 0.001313 | 100% |
| 0.10 | 1.0994 | 0.9769 | 0.9964 | 0.005227 | 100% |
| 0.20 | 1.1000 | 0.9371 | 0.9873 | 0.0194 | 100% |
| Max steps | 1 | 2 | 3 | 5 | 10 | 15 | 20 |
|---|---|---|---|---|---|---|---|
| Success (%) | 5.36 | 10.71 | 12.50 | 20.54 | 55.36 | 75.89 | 81.25 |
| Policy | Correct | Accuracy | vs ID | Repairs | Breaks |
|---|---|---|---|---|---|
| Identity | 201 | 31.854% | – | – | – |
| Old frozen Unified | 223 | 35.341% | 26 | 4 | |
| Runner-up + MEA v2 | 234 | 37.084% | 38 | 5 |
| Policy | Correct | Accuracy | vs ID | Repairs | Breaks |
|---|---|---|---|---|---|
| Identity | 197 | 31.270% | – | – | – |
| Old frozen Unified | 217 | 34.444% | 23 | 3 | |
| Runner-up + MEA v2 | 244 | 38.730% | 51 | 4 |
| Comparison | Peak accuracies | Difference | 95% CI |
|---|---|---|---|
| WHOLE MN | 61.6% vs 61.4% | ||
| WHOLE SA | 61.6% vs 61.0% | ||
| MN SA | 61.4% vs 61.0% |
| .25 | .50 | .75 | 1.00 | 1.25 | 1.50 | 1.75 | 2.00 | 2.25 | 2.50 | |
|---|---|---|---|---|---|---|---|---|---|---|
| MN | 36.88 | 40.00 | 45.00 | 54.38 | 56.88 | 59.38 | 61.25 | 63.75 | 63.75 | 60.63 |
| SA | 38.13 | 40.63 | 52.50 | 59.38 | 61.25 | 61.88 | 62.50 | 59.38 | 58.75 | 55.63 |
| WHOLE | 35.00 | 38.75 | 44.38 | 50.63 | 53.13 | 56.88 | 61.88 | 62.50 | 64.38 | 63.13 |
| Feature configuration | OOF accuracy |
|---|---|
| FULL | 67.50% |
| NO_INTERNAL | 66.875% |
| NO_OUTPUT_GEOMETRY | 65.625% |
| NO_PROPOSAL | 65.00% |
| BASELINE+PROPOSAL | 65.00% |
| OUTPUT_GEOM+PROPOSAL | 65.00% |
| Stage | Global MN | Router | Evidence class | ||
|---|---|---|---|---|---|
| SA-dev grouped OOF | 160 | 62.50% | 67.50% | Development / OOF | |
| Prospective frozen external | 156 | 66.67% | 71.15% | Frozen external | |
| Utility-final untouched | 294 | 60.54% | 66.33% | Confirmatory |
| Policy | Correct | Accuracy | Repairs | Breaks |
|---|---|---|---|---|
| Identity | 73 | 24.83% | – | – |
| Frozen global MN | 178 | 60.54% | 132 | 27 |
| Frozen Utility Router | 195 | 66.33% | 138 | 16 |
| Candidate oracle | 217 | 73.81% | – | – |
| Population | MN | Whole | 95% CI | ||
|---|---|---|---|---|---|
| Development OOF | 160 | 66.25% | 62.50% | ||
| Final diagnostic | 294 | 65.65% | 66.67% |
| Stage / policy | Correct | Accuracy | Repairs | Breaks | Oracle | |
|---|---|---|---|---|---|---|
| Dev low-rank restricted | 654 | 450 | 68.81% | 290 | 37 | 80.58% |
| Dev whole restricted | 654 | 450 | 68.81% | 287 | 34 | 80.58% |
| Dev unified | 654 | 470 | 71.87% | 305 | 32 | 83.18% |
| Final identity | 633 | 195 | 30.81% | – | – | – |
| Final whole restricted | 633 | 449 | 70.93% | 282 | 28 | 80.88% |
| Final unified | 633 | 463 | 73.14% | 298 | 30 | 84.36% |
| Head | Att. | Amp. | Strict | Mean shift (D/U) | Rep. | ||
|---|---|---|---|---|---|---|---|
| L33H1 | 85.33% | 86.00% | 76.67% | yes | |||
| L3H4 | 79.67% | 69.67% | 56.67% | yes | |||
| L18H13 | 91.67% | 92.67% | 89.00% | yes | |||
| L8H30 | 50.67% | 0.8625 | 47.00% | 0.8637 | 24.67% | no | |
| L3H11 | 64.00% | 57.67% | 0.0139 | 40.00% | yes | ||
| L13H11 | 71.33% | 72.67% | 49.67% | yes |
| Model / estimand | Raw Top-4 | Raw centered | Unit Top-4 | Unit centered |
|---|---|---|---|---|
| Llama L18H13 | 0.898229 | 0.867775 | 0.835115 | 0.650435 |
| IDEFICS B2H28 | 0.913626 | 0.839657 | 0.899015 | 0.784893 |
| Metric | Down | Up | Strict |
|---|---|---|---|
| MN-CSA consistency | 65.33% | 66.00% | 37.67% |
| Whole-head consistency | 85.33% | 88.00% | 74.00% |
| Mean MN-CSA shift | 0.01336 | 0.01269 | – |
| Mean whole-head shift | 0.03300 | 0.03702 | – |
| Effect retention | 40.47% | 34.28% | – |
| One-sided exact | – |
| Sens. share | Down cons. | Up cons. | Balanced shift | Down/Up retention | |
|---|---|---|---|---|---|
| 2 | 85.32% | 62% | 65% | 0.01289 | 40.69% / 40.50% |
| 4 | 91.36% | 63% | 68% | 0.01417 | 44.71% / 43.37% |
| 8 | 96.12% | 63% | 63% | 0.00898 | 28.35% / 39.73% |
| 16 | 98.36% | 61% | 61% | 0.00845 | 26.66% / 25.79% |
| Metric | Down | Up |
|---|---|---|
| MN directional consistency | 79.67% | 79.67% |
| Whole-head directional consistency | 84.00% | 84.33% |
| MN mean shift | 0.02555 | 0.02880 |
| Whole-head mean shift | 0.03151 | 0.03534 |
| Effect retention | 81.09% | 81.49% |
| Retention 95% CI | [71.21, 92.15]% | [71.90, 92.15]% |
| Oracle / subgroup | Value | Difference |
|---|---|---|
| MN oracle | 44.216% | vs identity |
| SA oracle | 40.095% | vs identity |
| Low-rank oracle (MN/SA) | 46.751% | vs identity |
| Whole oracle | 43.106% | vs identity |
| Unified oracle | 49.287% | vs identity |
| Complex ( ) | 32.444% 34.889% |
| 0.50 | 0.75 | 1.00 | 1.25 | 1.50 | 1.75 | 2.00 | Aggregate | |
| MN WHOLE break rate (pp) | +0.60 | -0.10 | -0.40 | -0.60 | -0.20 | -0.40 | -0.70 | -0.257 |
| Dataset | Bin | Mean | Repair | Break | |
|---|---|---|---|---|---|
| TallyQA | small | 267 | 0.04150 | 5.24% | 0.75% |
| TallyQA | medium | 266 | 0.16862 | 19.55% | 4.51% |
| TallyQA | large | 267 | 0.63573 | 23.97% | 11.61% |
| HoloCount | small | 294 | 0.06950 | 1.36% | 1.70% |
| HoloCount | medium | 293 | 0.31308 | 12.29% | 10.24% |
| HoloCount | large | 293 | 1.67425 | 10.92% | 12.97% |
| MN | SA | WHOLE | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gain | Accuracy | Mean | Repairs | Breaks | Accuracy | Mean | Repairs | Breaks | Accuracy | Mean | Repairs | Breaks |
| 1.0 | 50.8% | 0.000000 | 0 | 0 | 50.8% | 0.000000 | 0 | 0 | 50.8% | 0.000000 | 0 | 0 |
| 1.5 | 51.6% | -0.047078 | 12 | 8 | 51.4% | -0.107031 | 11 | 8 | 51.4% | -0.024922 | 13 | 10 |
| 2.0 | 52.0% | -0.072813 | 21 | 15 | 52.2% | -0.195453 | 20 | 13 | 50.8% | -0.063250 | 20 | 20 |
| 3.0 | 51.4% | -0.088031 | 40 | 37 | 51.2% | -0.336531 | 33 | 31 | 49.6% | -0.198234 | 32 | 38 |
| 4.0 | 54.2% | -0.068422 | 64 | 47 | 54.4% | -0.441625 | 57 | 39 | 49.4% | -0.395469 | 36 | 43 |
| Direction label | 1.0 | 1.5 | 2.0 | 3.0 | 4.0 | 6.0 | 8.0 |
|---|---|---|---|---|---|---|---|
| MN mean | 1.000 | 0.918 | 0.866 | 0.734 | 0.629 | 0.418 | 0.136 |
| MN median | 1.000 | 0.995 | 0.981 | 0.930 | 0.857 | 0.621 | 0.129 |
| SA mean | 1.000 | 0.910 | 0.855 | 0.737 | 0.621 | 0.407 | 0.146 |
| SA median | 1.000 | 0.995 | 0.981 | 0.944 | 0.892 | 0.716 | 0.296 |
| Prediction policy | APPLY correct | Full correct | Full accuracy |
|---|---|---|---|
| Unmodified identity | 5/63 | 5,207 | 65.1527% |
| Old Unified with matched gate | 25/63 | 5,227 | 65.4029% |
| Gate-conditional zero | 51/63 | 5,253 | 65.7282% |
| Frozen runner-up + MEA | 52/63 | 5,254 | 65.7407% |
| Policy / intervention scope | Global correct / 630 | APPLY correct / 63 |
|---|---|---|
| Identity (BF16, no steering) | 197 (31.270%) | 5 (7.937%) |
| Frozen AROMA old Unified | 217 (34.444%) | 25 (39.683%) |
| AROMA runner-up + MEA | 244 (38.730%) | 52 (82.540%) |
| GrAInS-inspired, counting-adapted, same-gate | 225 (35.714%) | 33 (52.381%) |
| GrAInS-inspired, counting-adapted, global | 247 (39.206%) | 33 (52.381%) |
| GrAInS minimal port, same-gate | 198 (31.429%) | 6 (9.524%) |
| Method | Localization / intervention object | Conditional control emphasis | Evaluation emphasis |
|---|---|---|---|
| NOTICE ( Golovanevsky et al., 2025 ) | causally localized multimodal components | diagnostic localization | corruption-based mechanism analysis |
| V-SEAM ( Wang et al., 2025 ) | causal visual-semantic components | intervention after attribution | editing / attention modulation |
| SteerVLM ( Sivakumar et al., 2025 ) | hidden activation dimensions | lightweight adaptive steering | robustness / off-target behavior |
| CAST ( Lee et al., 2025 ) | steering vectors | conditional activation steering | condition-dependent deployment |
| Rudman et al. ( Rudman et al., 2026 ) | hallucination-relevant mechanisms | causal intervention | prompt-induced hallucination analysis |
| VIB-Probe ( Zhang et al., 2026 ) | probe-derived hallucination signal | mitigation after detection | hallucination correction |
| Experiment | Baseline | Post/Oracle | ||
|---|---|---|---|---|
| Synthetic v2 OOF router | 1000 | 50.30 | 58.60 | +8.30 |
| Frozen v3 whole-head confirmation | 2000 | 49.95 | 58.05 | +8.10 |
| Final untouched v4 MN-CSA | 2000 | 50.25 | 58.25 | +8.00 |
| v4 whole-head reference | 2000 | 50.25 | 57.25 | +7.00 |
| Utility Router OOF (upward) | 160 | 62.50 | 67.50 | +5.00 |
| Utility Router frozen external (upward) | 156 | 66.67 | 71.15 | +4.487 |
| Comparison | Reference | Alternative | Evidence class | ||
|---|---|---|---|---|---|
| Utility Router final | 294 | Global MN 60.54 | Low-rank router 66.33 | Untouched confirmatory | |
| Adaptive whole challenge | 294 | Low-rank 66.33 | Whole 70.75 | Post-hoc diagnostic | |
| Matched budget OOF | 160 | MN 66.25 | Whole 62.50 | Development diagnostic | |
| Matched budget final | 294 | MN 65.65 | Whole 66.67 | Post-hoc diagnostic | |
| Procedural UGR development OOF | 654 | Whole restricted 68.81 | Unified 71.87 | Development / OOF | |
| Procedural UGR untouched final | 633 | Whole restricted 70.93 | Unified 73.14 | Prospective confirmatory |
| Experiment | Endpoint | Result | |
|---|---|---|---|
| E1 local response | 160 | local calibration | : Pearson .9936, Spearman .9990, sign 100%; median |
| Oracle MEA | 112 | strict target reachability | 91/112=81.25% within 20 steps |
| Frozen-direction matched budget | 112 | target reachability | 17/112=15.18%; MEA-only 74, frozen-only 0 |
| Activation-aligned matched budget | 112 | target reachability | 1/112=0.89%; MEA-only 90, activation-only 0 |
| Deployable v1 forced target | 623 targetable | assigned-target reachability / utility | 66.93% target reached but archived utility; 17 repairs / 134 breaks |
| Runner-up target dev | 30 APPLY | target quality | 24/30=80.0% target=GT; 15/16=93.75% among old repairs |
| Experiment | Primary endpoint | Result |
|---|---|---|
| Llama D1 multi-actuator ( ) | Bidirectional replication after Holm across 14 directional tests | 6/7 heads replicate; 5/6 non-primary; L8H30 is the negative case |
| GQA off-target ( ) | Aggregate paired break burden across seven matched budgets | WHOLE 1.4429%, MN 1.1857%; MN WHOLE , 95% CI ; sign-flip |
| Stage | Down cons. | Up cons. | Mean-effect retention (D/U) | |
|---|---|---|---|---|
| Held-out whole-head actuator confirmation | 300 | 82.00% | 82.67% | – |
| Equal-norm rank-4 MN-CSA | 300 | 65.33% | 66.00% | 40.47% / 34.28% |
| Dev-selected (dev200) | 200 | 73.00% | 74.00% | 73.69% / 73.62% |
| Untouched final | 300 | 79.67% | 79.67% | 81.09% / 81.49% |