Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecars

Daniel Henel; Frederik Werner; Alexander Langmann; Johannes Betz

cs.LGSep 17, 2026

Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecars

Authors: Daniel Henel, Frederik Werner, Alexander Langmann, Johannes Betz

Organizations: D. Henel, A. Langmann and J. Betz are with the Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI) · F. Werner is with the Institute of Automotive Technology, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI)

Abstract

Recent advances in large language models have improved their effectiveness as back-end components for voice assistants, particularly in intent understanding and context-aware input classification. However, online-hosted models introduce network dependency and variable inference latency, limiting their suitability for time-critical autonomous driving applications. In this work, we address these issues by developing Jarvis, an offline voice assistant for high-level behavioral commands of autonomous vehicles. Its architecture integrates speech recognition and synthesis with natural language command classification into a lightweight, local framework. Jarvis core component is a text-to-command classifier, built using a domain-specific fine-tuning of the Mistral 7B model, demonstrating low-latency inference. Our experimental evaluation demonstrates that our solution outperforms larger online-hosted models, achieving 97.63 % intent recognition accuracy with an average processing latency of 1.39 s, making it well-suited for operations requiring quick response times. To support further research and fine-tuning, we provide an open-source implementation.

Figures & tables

Fig. 1: System Architecture: Jarvis is a voice assistant for human operators used to control a autonomous racecar via voice commands.

Fig. 2: Autonomous driving software modules.

Fig. 3: System architecture and process flow of Jarvis. After wake-word detection, the system performs local speech-to-text transcription, LLM-based command classification, and text-to-speech feedback. Confirmed commands are transmitted via ROS2 to the base station, where safety checks precede transmission on the autonomous racing vehicle.

Model	Accuracy [%]	Avg. Latency [s]
Online Models
GPT-4o-2024-11-20	76.47	3.38
GPT-4o-2024-08-06	74.12	4.16
o3-mini-2025-01-31	85.88	21.08
GPT-4-turbo-2024-04-09	81.18	15.73
GPT-3.5-turbo-0125	57.65	5.12

TABLE I: Initial benchmarking of online and local Models. Highlighted models were selected for further fine-tuning.

Model	Accuracy [%]	Avg. Inference [s]	Epochs	Grad Acc Steps	r	alpha
Locally Deployed Models
Mistral-7B	76.74	0.92	2	4	32	32
Llama3.1-8B	75.58	0.88	3	4	32	32
Llama3.2-3B	73.26	0.67	3	4	32	32

TABLE II: Initial Fine-Tuning for Basic Commands Best Configurations for Selected Models

Parameter	Value
Model Name	Mistral-7B-v0.3-bnb-4bit
Accuracy [%]	97.63
Avg. Inference [s]	1.39
Learning Rate	$5\times 10^{-5}$
Epochs	2
LoRA Rank ( $r$ )	32

TABLE III: Best model results and hyperparameter after second stage fine-tuning

On-Device Language Model Inference

On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.

See all 169 papers

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving
Aug 13, 2026Zekai Li, Yihao Liang, Hongfei Zhang +3Efficient VLA Model Inference Autonomous Driving
1UC San Diego · 2Princeton · 3Independent Researcher
CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving
Jul 5, 2026Zhaohong Liu, Hao Ye, Xianlin Zhang +1VLMs for Autonomous Driving Efficient VLM Inference
State Key Laboratory of Networking and Switching Technology · Beijing University of Posts and Telecommunications, Beijing, China · School of Digital Media & Design Arts
Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
Aug 31, 2026Tongfei Guo, Lili SuVLMs for Autonomous Driving Efficient VLA Model Inference
Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA.

Aug 13, 2026J/K move · Enter open · S save

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

Zekai Li, Yihao Liang, Hongfei Zhang, Jian Chen+2

1UC San Diego · 2Princeton · 3Independent Researcher

Efficient VLA Model Inference Autonomous Driving Vision-Language-Action Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: [email protected] shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4~~Hz to 6.6~~Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.

Abstract

On-Device Language Model Inference

Explore similar work

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving