Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

Lyes Saad Saoud; Oualid Doukhi; Ehsan Reihani; Saeed Sepasi; Deok Jin Lee; Moussa Ayyash; Reza Ghorbani

cs.LGSep 13, 2026

Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

Authors: Lyes Saad Saoud, Oualid Doukhi, Ehsan Reihani, Saeed Sepasi, Deok Jin Lee, Moussa Ayyash, Reza Ghorbani

Organizations: Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi UAE · Jeonbuk National University Deokjin-gu Jeonju-si Jeollabuk-do South Korea · California State University Bakersfield USA · Hawaii Natural Energy Institute Hawaii HI USA ·

Abstract

Vision-Language-Action (VLA) models are studied mainly in robotics, where visual observations and language instructions are mapped to physical actions. This paper introduces Energy Vision-Language-Action (EVLA), a controlled multimodal benchmark for intent-conditioned residential energy management. EVLA frames battery scheduling as a multimodal trajectory-prediction problem in which an RGB energy-field representation, a numerical operating state, and a natural-language objective are mapped to a 16-step battery-action trajectory generated by a finite-horizon sampling-based reference generator. Source windows are derived from public residential electrical-load data, while electricity price, battery state of charge, indoor temperature, and time of day are generated benchmark metadata. A hidden operating regime is encoded only through energy-field texture, enabling paired visual changes while the explicit numerical state is fixed. Crossing 439,203 retained base windows with three hidden regimes and five language objectives yields 6,588,045 multimodal instances. An initial study evaluates 36 configurations over three training seeds using fixed subsets of 5,000 training, 500 validation, and 500 test instances. In the MobileNet-family comparison, removing processed language increases trajectory mean-squared error from 0.3856 +/- 0.0039 to 0.8628 +/- 0.0001, whereas removing vision yields 0.3843 +/- 0.0013, comparable to the full model. The results show strong asymmetry in modality use: the processed-language pathway is strongly associated with prediction quality, while the current RGB pathway provides no aggregate error advantage. These results characterize the fixed pilot subset and executed protocol rather than full-benchmark training. EVLA provides a controlled setting for studying how semantic intent and latent context influence residential energy-action prediction.

Figures & tables

Figure 1: EVLA task formulation for residential energy management. A synthetic RGB energy-field observation, a three-variable numerical benchmark state, and a natural-language operating objective are mapped to a 16-step normalized battery-action trajectory. The controlled hidden regime is available to the predictor only through the texture of the energy-field representation.

Figure 2: EVLA dataset- and target-generation pipeline. Processed residential CSV records and base-sample JSON metadata are used to extract 300-sample operating windows. Each window is converted into an RGB energy-field representation of size $256\times 128\times 3$ , expanded across three hidden regimes and five language objectives, and paired with a 16-step reference battery-action trajectory generated by a sampling-based shooting reference trajectory generator.

Figure 3: EVLA input preprocessing and modality encoders. (a) The $256\times 128\times 3$ energy field is resized to $224\times 224$ , normalized, and encoded into 16 visual tokens of width 256; their mean is retained for FiLM conditioning. (b) The stored instruction is mapped by the loader to an internal objective, processed by the simple language encoder, and projected to a 256-dimensional embedding. (c) The evaluated numerical state contains normalized price, SOC, and normalized indoor temperature; hour of day is retained as metadata and is not included in $S_{i}$ .

Figure 4: Vision–language conditioning of the learned action tokens and construction of the Transformer input sequence. Each action token is modulated by the visual token at the corresponding horizon position and then receives the same language embedding. The final sequence contains 16 visual tokens, one language token, and 16 conditioned action tokens, for a total shape of $33\times 256$ . Only the final 16 positions are decoded.

Figure 5: Multimodal FiLM conditioning and Transformer sequence model. The FiLM context concatenates the 3-D numerical state, the 256-D mean visual embedding, and the 256-D language embedding, giving 515 dimensions. An MLP $515\rightarrow 256\rightarrow 512$ produces 256-D scale and shift vectors. The $33\times 256$ token sequence is processed by three identical Transformer blocks with four-head self-attention, an action-token causal mask, FiLM modulation, and an FFN of size $256\rightarrow 1024\rightarrow 256$ . Future action-token positions are masked, whereas visual and language context remain visible.

Figure 6: Action decoding and deterministic baseline. The final 16 action-token representations are processed by one shared residual head and converted to a bounded learned residual. In parallel, the numerical state and loader-inferred internal objective determine a non-learned baseline action that is repeated over the 16-step horizon and normalized by $P_{\mathrm{scale}}$ . The final normalized trajectory is obtained by adding the baseline and learned residual. The figure intentionally shows the baseline pathway structurally; the complete rule set is specified in Eq. ( 31 ).

Component	Quantity	Value
Input image	native RGB size	$256\times 128\times 3$
Visual preprocessing	network input	$224\times 224\times 3$
Trajectory horizon	$H$	16
Common latent width	$d$	256
Visual representation	$E_{i}^{v}$	$16\times 256$
Mean visual embedding	$\overline{e}_{i}^{v}$	256

Table 1: Key EVLA trajectory-prediction dimensions and fixed configuration values used in the reported architecture.

Backbone	MSE	MAE	Params (M)	Time (ms)
MobileNet	$0.3829\pm 0.0003$	0.485	4.44	0.495
SimCLR	$0.3834\pm 0.0005$	0.486	14.93	0.788
ResNet	$0.3835\pm 0.0004$	0.486	14.26	0.823
TinyCNN	$0.3846\pm 0.0013$	0.487	4.17	0.298
ConvNeXt	$0.3849\pm 0.0019$	0.487	19.36	1.282
Swin	$0.3854\pm 0.0024$	0.488	19.26	1.236

Table 2: Visual-backbone comparison aggregated over seeds 42, 43, and 44. Timing values are approximate per-instance diagnostics.

Variant	MSE	MAE	Energy proxy	Viol.
Full	$0.3856\pm 0.0039$	0.488	0.042	0.0000
No vision	$0.3843\pm 0.0013$	0.487	0.040	0.0000
No language	$0.8628\pm 0.0001$	0.721	0.088	0.0022
State only	$0.8626\pm 0.0000$	0.719	0.088	0.0023

Table 3: MobileNet modality ablation aggregated over seeds 42, 43, and 44. Energy and violation values are internal benchmark diagnostics.

Variant	MSE	Params (M)	Time (ms)	Lang. gap
Base only	$8.0795\pm 0.0000$	1.26	0.114	–
Full FiLM	$0.3922\pm 0.0111$	4.44	0.490	-0.927
MLP fusion	$0.3832\pm 0.0010$	1.46	0.453	-0.935
No residual	$0.3829\pm 0.0001$	4.44	0.490	-0.008

Table 4: Architecture ablation for MobileNet-based EVLA variants, aggregated over seeds 42, 43, and 44.

Panel of Figure 7 — Figure 7: Principal EVLA ablations aggregated over seeds 42, 43, and 44: (a) trajectory MSE for the six designated full-modality visual backbones; (b) MobileNet-family modality ablation, showing strong dependence of the implemented multimodal pathway on processed language but no aggregate degradation after removing vision; and (c) MobileNet architecture ablation on a logarithmic scale, showing the substantially larger error of the deterministic base-only model. The language result is architecture-specific and should be interpreted together with the numerical-only baselines in Table 5 .

Model	MSE	MAE	Params (M)	Time (ms)	Energy proxy
MobileNet	$0.3829\pm 0.0003$	0.4850	4.44	0.495	0.0347
MLP	$0.3887\pm 0.0001$	0.4905	0.07	0.008	0.0402
LSTM	$0.3893\pm 0.0004$	0.4913	1.06	0.134	0.0423

Table 5: Comparison of the designated MobileNet entry with the numerical neural baselines, aggregated over seeds 42, 43, and 44. Timing values are approximate, and energy values are internal benchmark proxies.

Variant	MSE	MAE	RMSE	Params (M)	Time (ms)	Lang. gap
Backbone-Mobilenet	0.3829 $\pm$ 0.0003	0.485	0.619	4.44	0.495	-0.893
Arch-NoResidual-MobileNet	0.3829 $\pm$ 0.0001	0.485	0.619	4.44	0.490	-0.008
Arch-MLPFusion-MobileNet	0.3832 $\pm$ 0.0010	0.486	0.619	1.46	0.453	-0.935
Modality-NoVision-Resnet	0.3833 $\pm$ 0.0006	0.486	0.619	3.33	0.186	-0.939
Modality-Full-Convnext	0.3834 $\pm$ 0.0010	0.486	0.619	19.36	1.289	-0.922
Backbone-Simclr	0.3834 $\pm$ 0.0005	0.486	0.619	14.93	0.788	-0.929

Table 6: Ten named configurations with the lowest mean MSE in the aggregated ablation study.

Metric	Value
Total multimodal samples	6,588,045
Unique base operating states	439,203
Expansion factor	15
Samples per $(Z,L)$ cell	439,203
Malformed JSON files	0
Missing state files	0

Table S1 : Detailed EVLA dataset validation summary.

Figure S1 : Representative EVLA energy-field observations and corresponding 16-step action trajectories.

Figure S2 : Diagnostic used to inspect the paired multimodal construction.

Variant	MSE	MAE	RMSE	Params (M)	Inf. (ms)	Energy	Viol.
Backbone-Mobilenet	0.3829 $\pm$ 0.0003	0.4850 $\pm$ 0.0007	0.6188 $\pm$ 0.0002	4.44 $\pm$ 0.00	0.495 $\pm$ 0.004	0.0347 $\pm$ 0.0035	0.0000 $\pm$ 0.0000
Arch-NoResidual-MobileNet	0.3829 $\pm$ 0.0001	0.4851 $\pm$ 0.0001	0.6188 $\pm$ 0.0001	4.44 $\pm$ 0.00	0.490 $\pm$ 0.003	0.0392 $\pm$ 0.0002	0.0000 $\pm$ 0.0000
Arch-MLPFusion-MobileNet	0.3832 $\pm$ 0.0010	0.4856 $\pm$ 0.0018	0.6190 $\pm$ 0.0008	1.46 $\pm$ 0.00	0.453 $\pm$ 0.014	0.0382 $\pm$ 0.0063	0.0000 $\pm$ 0.0000
Modality-NoVision-Resnet	0.3833 $\pm$ 0.0006	0.4857 $\pm$ 0.0004	0.6191 $\pm$ 0.0005	3.33 $\pm$ 0.00	0.186 $\pm$ 0.007	0.0384 $\pm$ 0.0011	0.0000 $\pm$ 0.0000
Modality-Full-Convnext	0.3834 $\pm$ 0.0010	0.4857 $\pm$ 0.0012	0.6192 $\pm$ 0.0008	19.36 $\pm$ 0.00	1.289 $\pm$ 0.013	0.0391 $\pm$ 0.0038	0.0000 $\pm$ 0.0000
Backbone-Simclr	0.3834 $\pm$ 0.0005	0.4856 $\pm$ 0.0005	0.6192 $\pm$ 0.0004	14.93 $\pm$ 0.00	0.788 $\pm$ 0.002	0.0377 $\pm$ 0.0017	0.0000 $\pm$ 0.0000

Table S2 : Complete 36-configuration ablation results aggregated over seeds 42, 43, and 44.

Run	MSE	MAE	RMSE	Energy	Viol.	Inf. (ms)	Params (M)
seed42-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.113	1.26
seed43-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.108	1.26
seed44-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.119	1.26
seed42-Arch-Full-MobileNet	0.4047	0.5017	0.6362	0.0496	0.0000	0.490	4.44
seed43-Arch-Full-MobileNet	0.3833	0.4856	0.6191	0.0381	0.0000	0.479	4.44
seed44-Arch-Full-MobileNet	0.3887	0.4911	0.6234	0.0521	0.0000	0.502	4.44

Table S3 : Complete seed-level results for the 108 reported runs.

Component	Configuration
Operating system	Ubuntu 22.04.5 LTS
CPU	Intel Core Ultra 9 275HX
Logical CPUs	24
System memory	62 GiB RAM
GPU	NVIDIA GeForce RTX 5090 GPU
GPU memory	24 GB

Table S4 : Hardware and software environment used for the EVLA experiments.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design
Oct 4, 2026Seonghun Jung, Sieun Moon, Jiyoung Jeong +2Efficient VLA Model Inference Vision-Language-Action Models
KAIST, Daejeon, Republic of Korea
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
Mar 14, 2026Suhwan Choi, Yunsung Lee, Yubeen Park +4VLM Evaluation Vision-Language-Action Models
Allen Institute for AI (AI2) · Seoul National University

Mar 19, 2026J/K move · Enter open · S save

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Bryce Grant, Xijia Zhao, Peng Wang

Case Western Reserve University

Vision-Language-Action Models Robotics

Open paper AskPDF arXiv

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (

n{=}1{,}968

). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Abstract

Vision-Language-Action Models

Explore similar work

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models