A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collection

Philipp Reis; Philipp Rigoll; Martin Zehetner; Jacqueline Henle; Stefan Otten; Eric Sax

cs.DBMar 13, 2026

A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collection

Authors: Philipp Reis, Philipp Rigoll, Martin Zehetner, Jacqueline Henle, Stefan Otten, Eric Sax

Organizations: FZI Research Center for Information Technology, Karlsruhe, Germany

Abstract

Data-driven systems depend on task-relevant data, yet data collection pipelines remain passive and indiscriminate. Continuous logging of multimodal sensor streams incurs high storage costs and captures irrelevant data. This paper proposes a declarative framework for intent-driven, on-device data collection that enables selective collection of multimodal sensor data based on high-level user requests. The framework combines natural language interaction with a formally specified domain-specific language (DSL). Large language models translate user-defined requirements into verifiable and composable DSL programs that define conditional triggers across heterogeneous sensors, including cameras, LiDAR, and system telemetry. Empirical evaluation on vehicular and robotic perception tasks shows that the DSL-based approach achieves higher generation consistency and lower execution latency than unconstrained code generation while maintaining comparable detection performance. The structured abstraction supports modular trigger composition and concurrent deployment on resource-constrained edge platforms. This approach replaces passive logging with a verifiable, intent-driven mechanism for multimodal data collection in real-time systems.

Figures & tables

Work	Modulare Framework	Multimodality	Declarative	Online Selection
SOLA [ 1 ]	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$
Uniqueness [ 2 ]	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$
CERN [ 3 ]	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$
ScenarioRetrieval [ 4 ]	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$
ViperGPT [ 5 ]	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$
RegEx [ 6 ]	$\Large\color[rgb]{0,1,0}{\checkmark}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{1,0,0}{\boldsymbol{\times}}$	$\Large\color[rgb]{0,1,0}{\checkmark}$

TABLE I: Comparison of Triggering and Data Collection Frameworks.

Fig. 1: Proposed framework for DSL-based trigger generation. Given a data query, available tools, and system information, the framework generates a structured trigger configuration, which is deployed to the vehicle trigger architecture for selective data collection.

Panel of Figure 3 — Fig. 2: Definition of the DSL using the extended Backus-Naur form (EBNF).

DSL Construct	JSON Representation	Description
TRIGGER <id> (frequency=2) {<body>}	{"type": "trigger", "name": "<id>", "spec": {"frequency": 2}, ...}	Trigger-level sampling specification (e.g., evaluation rate in Hz)
PREDICATE(detect, class="person")	{"type": "detect", "target": {"class": "person"}}	Detect object of given class
PREDICATE(<type>, on=<x>, ...)	Add "on": "<x>" to JSON	Predicate input reference (topic name / stored predicate result)
PREDICATE(count, on="p", operator=">", value=5)	{"type": "count", "on": "p", "comparison": {"operator": ">", "value": 5}}	Count comparison on variable
PREDICATE(... AS x)	Add "store_as": "x" to JSON	Stores result as variable
NOT PREDICATE(...)	Add "not": true to JSON	Logical negation (optional)

TABLE II: Mapping from EBNF defined DSL to JSON-based representation.

Fig. 3: Workflow for scalable data collection. Data requests are processed by a LLMs to a JSON files based on the proposed DSL. Each JSON File is the configuration for an individual trigger, which is the interface for the trigger framework.

Fig. 4: Data flow within the vehicle system for trigger-based data collection. Data are published to system topics, preprocessed by the Data Manager, and cached for trigger evaluation before being forwarded to the Trigger Manager. The Trigger Manager evaluates each trigger according to its configuration. Upon activation, the corresponding data cache is persisted.

Fig. 5: Three exemplary triggers generated by the proposed framework. A data query (left) is transformed into atomic predicates (middle), which are composed into a detection logic and evaluated against the current data cache (right).

Single Query	Reasoning Query	Conditional Query
I want to find images with stop signs.	I need data with persons in front of the recording car.	I want data with a red traffic light.
I need situations with trains.	I need situations with a person riding a bike.	I need data with more than 5 cars.
I need data when it’s snowy.	I need data involving car-following scenarios.	Find me pedestrians when its snowy.
Find me situations with a minimum of 60% brightness.	Find me images where the car in front is less than 5m away.	Find me red cars.
I want data with persons visible.		Find me situations with a person and a car.

TABLE III: Data queries grouped by category.

Fig. 6: Comparison of detection performance for data-triggered events across the DSL framework, the code generation approach, and the VLM-based approach.

	Sequence	Levenshtein	Cosine
DSL	$\bm{0.966\pm 0.067}$	$\bm{0.955\pm 0.106}$	$\bm{0.918\pm 0.111}$
Code	$0.392\pm 0.225$	$0.510\pm 0.207$	$0.802\pm 0.134$

TABLE IV: Comparision of the consistency of generated triggers between the proposed trigger framework and the plain code generation.s

Fig. 7: Execution runtime comparison between the proposed framework, the VLM-based approach, and the plain code detection approach.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 178 papers

Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection
Jun 25, 2026Bo ChenConstrained Generative Modeling Web Agents
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams
Jun 19, 2026Cong Wan, Zeyu Guo, Zijian Cai +6Training Data Curation Synthetic Data Generation for Language Models
Xi’an Jiaotong University · University of Chinese Academy of Sciences · Shenzhen University of Advanced Technology +1
SensorPersona: An LLM-Empowered System for Continual Persona Extraction from Longitudinal Mobile Sensor Streams
Mar 15, 2026Bufang Yang, Lilin Xu, Yixuan Li +3User Modeling LLM Personalization
The Chinese University of Hong Kong, Hong Kong SAR · Columbia University, United States

Jun 25, 2026J/K move · Enter open · S save

Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

Bo Chen

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Constrained Generative Modeling Web Agents LLM Agent Reliability

Open paper AskPDF arXiv

LLMs and agents can generate web scrapers from natural-language requirements, but direct generation remains unreliable because of dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection.

Abstract

Robotic Data Collection

Explore similar work

Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

SensorPersona: An LLM-Empowered System for Continual Persona Extraction from Longitudinal Mobile Sensor Streams