Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

Lyes Saad Saoud; Oualid Doukhi; Ehsan Reihani; Saeed Sepasi; Deok Jin Lee; Moussa Ayyash; Reza Ghorbani

cs.LGSep 13, 2026

Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

Authors: Lyes Saad Saoud, Oualid Doukhi, Ehsan Reihani, Saeed Sepasi, Deok Jin Lee, Moussa Ayyash, Reza Ghorbani

Organizations: Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi UAE · Jeonbuk National University Deokjin-gu Jeonju-si Jeollabuk-do South Korea · California State University Bakersfield USA · Hawaii Natural Energy Institute Hawaii HI USA ·

Abstract

Vision-Language-Action (VLA) models are studied mainly in robotics, where visual observations and language instructions are mapped to physical actions. This paper introduces Energy Vision-Language-Action (EVLA), a controlled multimodal benchmark for intent-conditioned residential energy management. EVLA frames battery scheduling as a multimodal trajectory-prediction problem in which an RGB energy-field representation, a numerical operating state, and a natural-language objective are mapped to a 16-step battery-action trajectory generated by a finite-horizon sampling-based reference generator. Source windows are derived from public residential electrical-load data, while electricity price, battery state of charge, indoor temperature, and time of day are generated benchmark metadata. A hidden operating regime is encoded only through energy-field texture, enabling paired visual changes while the explicit numerical state is fixed. Crossing 439,203 retained base windows with three hidden regimes and five language objectives yields 6,588,045 multimodal instances. An initial study evaluates 36 configurations over three training seeds using fixed subsets of 5,000 training, 500 validation, and 500 test instances. In the MobileNet-family comparison, removing processed language increases trajectory mean-squared error from 0.3856 +/- 0.0039 to 0.8628 +/- 0.0001, whereas removing vision yields 0.3843 +/- 0.0013, comparable to the full model. The results show strong asymmetry in modality use: the processed-language pathway is strongly associated with prediction quality, while the current RGB pathway provides no aggregate error advantage. These results characterize the fixed pilot subset and executed protocol rather than full-benchmark training. EVLA provides a controlled setting for studying how semantic intent and latent context influence residential energy-action prediction.

Figures & tables

Figure 1: EVLA task formulation for residential energy management. A synthetic RGB energy-field observation, a three-variable numerical benchmark state, and a natural-language operating objective are mapped to a 16-step normalized battery-action trajectory. The controlled hidden regime is available to the predictor only through the texture of the energy-field representation.

Figure 2: EVLA dataset- and target-generation pipeline. Processed residential CSV records and base-sample JSON metadata are used to extract 300-sample operating windows. Each window is converted into an RGB energy-field representation of size $256\times 128\times 3$ , expanded across three hidden regimes and five language objectives, and paired with a 16-step reference battery-action trajectory generated by a sampling-based shooting reference trajectory generator.

Figure 3: EVLA input preprocessing and modality encoders. (a) The $256\times 128\times 3$ energy field is resized to $224\times 224$ , normalized, and encoded into 16 visual tokens of width 256; their mean is retained for FiLM conditioning. (b) The stored instruction is mapped by the loader to an internal objective, processed by the simple language encoder, and projected to a 256-dimensional embedding. (c) The evaluated numerical state contains normalized price, SOC, and normalized indoor temperature; hour of day is retained as metadata and is not included in $S_{i}$ .

Figure 4: Vision–language conditioning of the learned action tokens and construction of the Transformer input sequence. Each action token is modulated by the visual token at the corresponding horizon position and then receives the same language embedding. The final sequence contains 16 visual tokens, one language token, and 16 conditioned action tokens, for a total shape of $33\times 256$ . Only the final 16 positions are decoded.

Figure 5: Multimodal FiLM conditioning and Transformer sequence model. The FiLM context concatenates the 3-D numerical state, the 256-D mean visual embedding, and the 256-D language embedding, giving 515 dimensions. An MLP $515\rightarrow 256\rightarrow 512$ produces 256-D scale and shift vectors. The $33\times 256$ token sequence is processed by three identical Transformer blocks with four-head self-attention, an action-token causal mask, FiLM modulation, and an FFN of size $256\rightarrow 1024\rightarrow 256$ . Future action-token positions are masked, whereas visual and language context remain visible.

Figure 6: Action decoding and deterministic baseline. The final 16 action-token representations are processed by one shared residual head and converted to a bounded learned residual. In parallel, the numerical state and loader-inferred internal objective determine a non-learned baseline action that is repeated over the 16-step horizon and normalized by $P_{\mathrm{scale}}$ . The final normalized trajectory is obtained by adding the baseline and learned residual. The figure intentionally shows the baseline pathway structurally; the complete rule set is specified in Eq. ( 31 ).

Component	Quantity	Value
Input image	native RGB size	$256\times 128\times 3$
Visual preprocessing	network input	$224\times 224\times 3$
Trajectory horizon	$H$	16
Common latent width	$d$	256
Visual representation	$E_{i}^{v}$	$16\times 256$
Mean visual embedding	$\overline{e}_{i}^{v}$	256

Table 1: Key EVLA trajectory-prediction dimensions and fixed configuration values used in the reported architecture.

Backbone	MSE	MAE	Params (M)	Time (ms)
MobileNet	$0.3829\pm 0.0003$	0.485	4.44	0.495
SimCLR	$0.3834\pm 0.0005$	0.486	14.93	0.788
ResNet	$0.3835\pm 0.0004$	0.486	14.26	0.823
TinyCNN	$0.3846\pm 0.0013$	0.487	4.17	0.298
ConvNeXt	$0.3849\pm 0.0019$	0.487	19.36	1.282
Swin	$0.3854\pm 0.0024$	0.488	19.26	1.236

Table 2: Visual-backbone comparison aggregated over seeds 42, 43, and 44. Timing values are approximate per-instance diagnostics.

Variant	MSE	MAE	Energy proxy	Viol.
Full	$0.3856\pm 0.0039$	0.488	0.042	0.0000
No vision	$0.3843\pm 0.0013$	0.487	0.040	0.0000
No language	$0.8628\pm 0.0001$	0.721	0.088	0.0022
State only	$0.8626\pm 0.0000$	0.719	0.088	0.0023

Table 3: MobileNet modality ablation aggregated over seeds 42, 43, and 44. Energy and violation values are internal benchmark diagnostics.

Variant	MSE	Params (M)	Time (ms)	Lang. gap
Base only	$8.0795\pm 0.0000$	1.26	0.114	–
Full FiLM	$0.3922\pm 0.0111$	4.44	0.490	-0.927
MLP fusion	$0.3832\pm 0.0010$	1.46	0.453	-0.935
No residual	$0.3829\pm 0.0001$	4.44	0.490	-0.008

Table 4: Architecture ablation for MobileNet-based EVLA variants, aggregated over seeds 42, 43, and 44.

Panel of Figure 7 — Figure 7: Principal EVLA ablations aggregated over seeds 42, 43, and 44: (a) trajectory MSE for the six designated full-modality visual backbones; (b) MobileNet-family modality ablation, showing strong dependence of the implemented multimodal pathway on processed language but no aggregate degradation after removing vision; and (c) MobileNet architecture ablation on a logarithmic scale, showing the substantially larger error of the deterministic base-only model. The language result is architecture-specific and should be interpreted together with the numerical-only baselines in Table 5 .

Model	MSE	MAE	Params (M)	Time (ms)	Energy proxy
MobileNet	$0.3829\pm 0.0003$	0.4850	4.44	0.495	0.0347
MLP	$0.3887\pm 0.0001$	0.4905	0.07	0.008	0.0402
LSTM	$0.3893\pm 0.0004$	0.4913	1.06	0.134	0.0423

Table 5: Comparison of the designated MobileNet entry with the numerical neural baselines, aggregated over seeds 42, 43, and 44. Timing values are approximate, and energy values are internal benchmark proxies.

Variant	MSE	MAE	RMSE	Params (M)	Time (ms)	Lang. gap
Backbone-Mobilenet	0.3829 $\pm$ 0.0003	0.485	0.619	4.44	0.495	-0.893
Arch-NoResidual-MobileNet	0.3829 $\pm$ 0.0001	0.485	0.619	4.44	0.490	-0.008
Arch-MLPFusion-MobileNet	0.3832 $\pm$ 0.0010	0.486	0.619	1.46	0.453	-0.935
Modality-NoVision-Resnet	0.3833 $\pm$ 0.0006	0.486	0.619	3.33	0.186	-0.939
Modality-Full-Convnext	0.3834 $\pm$ 0.0010	0.486	0.619	19.36	1.289	-0.922
Backbone-Simclr	0.3834 $\pm$ 0.0005	0.486	0.619	14.93	0.788	-0.929

Table 6: Ten named configurations with the lowest mean MSE in the aggregated ablation study.

Metric	Value
Total multimodal samples	6,588,045
Unique base operating states	439,203
Expansion factor	15
Samples per $(Z,L)$ cell	439,203
Malformed JSON files	0
Missing state files	0

Table S1 : Detailed EVLA dataset validation summary.

Figure S1 : Representative EVLA energy-field observations and corresponding 16-step action trajectories.

Figure S2 : Diagnostic used to inspect the paired multimodal construction.

Variant	MSE	MAE	RMSE	Params (M)	Inf. (ms)	Energy	Viol.
Backbone-Mobilenet	0.3829 $\pm$ 0.0003	0.4850 $\pm$ 0.0007	0.6188 $\pm$ 0.0002	4.44 $\pm$ 0.00	0.495 $\pm$ 0.004	0.0347 $\pm$ 0.0035	0.0000 $\pm$ 0.0000
Arch-NoResidual-MobileNet	0.3829 $\pm$ 0.0001	0.4851 $\pm$ 0.0001	0.6188 $\pm$ 0.0001	4.44 $\pm$ 0.00	0.490 $\pm$ 0.003	0.0392 $\pm$ 0.0002	0.0000 $\pm$ 0.0000
Arch-MLPFusion-MobileNet	0.3832 $\pm$ 0.0010	0.4856 $\pm$ 0.0018	0.6190 $\pm$ 0.0008	1.46 $\pm$ 0.00	0.453 $\pm$ 0.014	0.0382 $\pm$ 0.0063	0.0000 $\pm$ 0.0000
Modality-NoVision-Resnet	0.3833 $\pm$ 0.0006	0.4857 $\pm$ 0.0004	0.6191 $\pm$ 0.0005	3.33 $\pm$ 0.00	0.186 $\pm$ 0.007	0.0384 $\pm$ 0.0011	0.0000 $\pm$ 0.0000
Modality-Full-Convnext	0.3834 $\pm$ 0.0010	0.4857 $\pm$ 0.0012	0.6192 $\pm$ 0.0008	19.36 $\pm$ 0.00	1.289 $\pm$ 0.013	0.0391 $\pm$ 0.0038	0.0000 $\pm$ 0.0000
Backbone-Simclr	0.3834 $\pm$ 0.0005	0.4856 $\pm$ 0.0005	0.6192 $\pm$ 0.0004	14.93 $\pm$ 0.00	0.788 $\pm$ 0.002	0.0377 $\pm$ 0.0017	0.0000 $\pm$ 0.0000

Table S2 : Complete 36-configuration ablation results aggregated over seeds 42, 43, and 44.

Run	MSE	MAE	RMSE	Energy	Viol.	Inf. (ms)	Params (M)
seed42-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.113	1.26
seed43-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.108	1.26
seed44-Arch-BaseOnly-MobileNet	8.0795	2.6864	2.8425	1.0925	0.1084	0.119	1.26
seed42-Arch-Full-MobileNet	0.4047	0.5017	0.6362	0.0496	0.0000	0.490	4.44
seed43-Arch-Full-MobileNet	0.3833	0.4856	0.6191	0.0381	0.0000	0.479	4.44
seed44-Arch-Full-MobileNet	0.3887	0.4911	0.6234	0.0521	0.0000	0.502	4.44

Table S3 : Complete seed-level results for the 108 reported runs.

Component	Configuration
Operating system	Ubuntu 22.04.5 LTS
CPU	Intel Core Ultra 9 275HX
Logical CPUs	24
System memory	62 GiB RAM
GPU	NVIDIA GeForce RTX 5090 GPU
GPU memory	24 GB

Table S4 : Hardware and software environment used for the EVLA experiments.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

Mar 19, 2026

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (

n{=}1{,}968

). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Bryce Grant, Xijia Zhao, Peng Wang

Case Western Reserve University

Vision-Language-Action Models Robotics

Ask PDF

Oct 4, 2026

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design

Vision-language-action (VLA) models translate multimodal observations into low-level robot actions. During robot operation, each control period sets an inference deadline, and overruns leave the robot acting on stale observations, reducing task success. Meeting this deadline motivates on-device or nearby edge execution, where a single robot requires batch-1 inference outside the design point of LLM serving systems. Although VLA architectures combine familiar vision-language, autoregressive, and diffusion-style components, their runtime behavior in this batch-1 control setting remains uncharacterized. We characterize four representative VLA models on an edge GPU server and two onboard SoCs, using single-inference profiling and 43,200 closed-loop episodes. Action tensor dimensionality determines whether a stage is memory- or compute-bound, platform balance can shift that bottleneck, and GPU frequency scaling yields a platform-dependent energy-latency sweet spot. In closed-loop operation, overlapping inference with action execution creates an accuracy-speed-energy tradeoff, and no configuration is Pareto-dominant across deployment SLOs. These results guide joint design of VLA model architectures, hardware, and runtime policies.

Seonghun Jung, Sieun Moon, Jiyoung Jeong +2

KAIST, Daejeon, Republic of Korea

Efficient VLA Model Inference Vision-Language-Action Models VLMs for Robotics

Ask PDF

Mar 14, 2026

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pipeline requires resolving incompatible dependencies, matching underspecified evaluation protocols, and reverse-engineering undocumented preprocessing. This burden scales with the number of models and benchmarks, making comprehensive evaluation impractical for most teams. We present vla-eval, an open-source evaluation harness that eliminates this per-benchmark cost by decoupling model inference from benchmark execution through a WebSocket+msgpack protocol with Docker-based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four-method interface; the full cross-evaluation matrix works automatically. The framework supports 14 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes. To validate the framework, we reproduce published scores across six VLA codebases and three benchmarks, documenting previously undocumented pitfalls. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available at https://github.com/allenai/vla-evaluation-harness and https://allenai.github.io/vla-evaluation-harness/leaderboard.

Suhwan Choi, Yunsung Lee, Yubeen Park +4

Allen Institute for AI (AI2) · Seoul National University

VLM Evaluation Vision-Language-Action Models

Ask PDF

Abstract

Vision-Language-Action Models

Explore similar work

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models