A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collection
Organizations: FZI Research Center for Information Technology, Karlsruhe, Germany
Abstract
Data-driven systems depend on task-relevant data, yet data collection pipelines remain passive and indiscriminate. Continuous logging of multimodal sensor streams incurs high storage costs and captures irrelevant data. This paper proposes a declarative framework for intent-driven, on-device data collection that enables selective collection of multimodal sensor data based on high-level user requests. The framework combines natural language interaction with a formally specified domain-specific language (DSL). Large language models translate user-defined requirements into verifiable and composable DSL programs that define conditional triggers across heterogeneous sensors, including cameras, LiDAR, and system telemetry. Empirical evaluation on vehicular and robotic perception tasks shows that the DSL-based approach achieves higher generation consistency and lower execution latency than unconstrained code generation while maintaining comparable detection performance. The structured abstraction supports modular trigger composition and concurrent deployment on resource-constrained edge platforms. This approach replaces passive logging with a verifiable, intent-driven mechanism for multimodal data collection in real-time systems.
Figures & tables
| Work | Modulare Framework | Multimodality | Declarative | Online Selection |
|---|---|---|---|---|
| SOLA [ 1 ] | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} |
| Uniqueness [ 2 ] | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} |
| CERN [ 3 ] | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} |
| ScenarioRetrieval [ 4 ] | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} |
| ViperGPT [ 5 ] | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} |
| RegEx [ 6 ] | \Large\color[rgb]{0,1,0}{\checkmark} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{1,0,0}{\boldsymbol{\times}} | \Large\color[rgb]{0,1,0}{\checkmark} |
| DSL Construct | JSON Representation | Description |
|---|---|---|
| TRIGGER <id> (frequency=2) {<body>} | {"type": "trigger", "name": "<id>", "spec": {"frequency": 2}, ...} | Trigger-level sampling specification (e.g., evaluation rate in Hz) |
| PREDICATE(detect, class="person") | {"type": "detect", "target": {"class": "person"}} | Detect object of given class |
| PREDICATE(<type>, on=<x>, ...) | Add "on": "<x>" to JSON | Predicate input reference (topic name / stored predicate result) |
| PREDICATE(count, on="p", operator=">", value=5) | {"type": "count", "on": "p", "comparison": {"operator": ">", "value": 5}} | Count comparison on variable |
| PREDICATE(... AS x) | Add "store_as": "x" to JSON | Stores result as variable |
| NOT PREDICATE(...) | Add "not": true to JSON | Logical negation (optional) |
| Single Query | Reasoning Query | Conditional Query |
|---|---|---|
| I want to find images with stop signs. | I need data with persons in front of the recording car. | I want data with a red traffic light. |
| I need situations with trains. | I need situations with a person riding a bike. | I need data with more than 5 cars. |
| I need data when it’s snowy. | I need data involving car-following scenarios. | Find me pedestrians when its snowy. |
| Find me situations with a minimum of 60% brightness. | Find me images where the car in front is less than 5m away. | Find me red cars. |
| I want data with persons visible. | Find me situations with a person and a car. |
| Sequence | Levenshtein | Cosine | |
|---|---|---|---|
| DSL | |||
| Code |
Robotic Data Collection
Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.
See all 178 papers