cs.AIOct 22, 2026

AgentDP: Towards Automated Model Deployment for Heterogeneous Embedded AI Applications

Authors: Leming Shen, Qiang Yang, Xinyu Huang, Zijing Ma, Yuanqing Zheng

Organizations: The Hong Kong Polytechnic University · University College London · U2University of Cambridge

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper presents AgentDP, which aims to deploy pre-trained AI models on heterogeneous embedded devices by fully automating model optimization and adaptation (O&A) with Large Language Model (LLM) agents. AgentDP not only significantly reduces human effort via automation but also enhances model inference efficacy by integrating LLMs with device-specific domain knowledge. To deliver a robust automation framework, AgentDP develops novel solutions to address three key technical challenges: 1) To help users elicit clear requirements, we propose a user requirement standardization module; 2) To mitigate the execution environment mismatch between AI models and heterogeneous embedded devices, we design an operator-aware model partition paradigm that enables different parts of the model to be efficiently executed on distinct inference engines. 3) To avoid re-searching optimization hyperparameters for new tasks, we propose a configuration reuse module that heuristically retrieves previous hyperparameter configurations for efficient reuse. Extensive experiments demonstrate that by fully automating the entire O&A process, AgentDP reduces entire model adaptation time by up to 80% compared with manual O&A while achieving higher task accuracy (more than 13% on average) than the baselines. Additionally, a comprehensive user study also exhibits the robust generalizability of AgentDP in adapting to different user requirements for heterogeneous embedded devices.

Figures & tables

Figure 1. Deploying pre-trained open-source models on embedded devices (manual vs. automated).
Figure 2. The mismatch between models and engines.
Figure 3. An illustrative workflow of LLM agents.
Figure 4. The system overview and workflow of AgentDP with four specialized agents.
Figure 5. The workflow of User Interface Agent.
Figure 6. (a) AgentDP prompts UIA to identify ambiguities and request the user to make clarification; (b) The standardized user requirements with clear structures.
Figure 7. The workflow of Hyperparameter Tuning Agent.
Figure 8. Example of ToT-based prompts.
Figure 9. The workflow of Model Adaptation Agent.
Figure 10. (a) A sample DAG with its operators from part of a portable model; (b) Our CEIP module.
Figure 11. The workflow of configuration reuse agent.
Figure 12. FSM for hyperparameter tuning modeling.
TypeDeviceRAMVRAMStorage
Embedded DeviceRaspberry Pi 58GB-32GB
Jetson Nano4GB4GB64GB
IBOX-606 mini PC32GB12GB256GB
RouterMi AX3000256MB--
Edge ServerSupermicro X11SCA-F32GB16GB1T
Table 1. Detailed information of the embedded devices.
Figure 13. Evaluation results of image recognition (ADP for AgentDP , M. for MapCoder, and O. for the original large model; Acc. for classification accuracy, Pre. for precision, Rec. for recall rate, and F1 for F1 score; the horizontal lines represent the user requirement (U.) for model deployment).
Figure 14. Evaluation results for speech sentiment analysis (SSA).
Figure 15. Evaluation results for vision-language understanding (In (c), A. for art & design, B. for business, S. for science, M. for medicine & healthcare, H. for humanity, and T. for technology & engineering).
Figure 16. Ablation study on image recognition (A1 for the original AgentDP , A2 for AgentDP without the URS module or CRA, and O. for the original large model).
CategoryBackground Information
GenderFemale (45%), Male (40%), Prefer not to say (15%)
AgeUnder 18 (5%), 18-30 (75%), 30-39 (10%), 40 and older (10%)
EducationBachelor (5%), Master (20%), Doctoral (70%), Others (5%)
EnglishBeginner (5%), Intermediate (20%), Advanced (65%), Fluent (10%)
ExpertiseExpert (50%), Semi-expert (50%)
Table 2. Participant information of user study (N=20)
Figure 17. User study.
More in this topic

MLOps

See all 11 papers

Explore similar work

CardsList
Apr 16, 2026

AIPC: Agent-Based Automation for AI Model Deployment with Qualcomm AI Runtime

Edge AI model deployment is a multi-stage engineering process involving model conversion, operator compatibility handling, quantization calibration, runtime integration, and accuracy validation. In practice, this workflow is long, failure-prone, and heavily dependent on deployment expertise, particularly when targeting hardware-specific inference runtimes. This technical report presents AIPC (AI Porting Conversion), an AI agent-driven approach for constrained automation of AI model deployment. AIPC decomposes deployment into standardized, verifiable stages and injects deployment-domain knowledge into agent execution through Agent Skills, helper scripts, and a stage-wise validation loop. This design reduces both the expertise barrier and the engineering time required for hardware deployment. Using Qualcomm AI Runtime (QAIRT) as the primary scenario, this report examines automated deployment across representative vision, multimodal, and speech models. In the cases covered here, AIPC can complete deployment from PyTorch to runnable QNN/SNPE inference within 7-20 minutes for structurally regular vision models, with indicative API costs roughly in the range of USD 0.7-10. For more complex models involving less-supported operators, dynamic shapes, or autoregressive decoding structures, fully automated deployment may still require further advances, but AIPC already provides practical support for execution, failure localization, and bounded repair.

Jianhao Su, Zhanwei Wu, ShengTing Huang +1

Qualcomm Technologies, Inc.

Edge ML
AskPDF
Jun 15, 2026

Embedded Arena: Iterative Optimization via Hardware Feedback

Embedded devices from wildlife monitoring stations to clinical wearables require local AI inference due to latency, communication, or privacy constraints. Optimizing models for heterogeneous microcontrollers (MCUs) requires simultaneously satisfying hard physical constraints on memory, power, and temperature while preserving accuracy, a multidimensional optimization that is today performed manually by experts. We ask whether an LLM agent can autonomously navigate this complex, multi-turn pipeline guided by real hardware feedback, and introduce a hardware-in-the-loop agent arena in which the agent iteratively refines both model and firmware -- compiling, flashing, and measuring on real hardware -- to enable closed-loop optimization. Frontier models, including Claude Opus 4.7 and Gemini 3.1 Pro, fail entirely without hardware feedback (0% deployment success), whereas our hardware-in-the-loop formulation achieves the first successful deployment within three iterations and can surpass human expert results within seven. This agentic co-optimization achieves 250x compression for vision models with <3.3% accuracy loss and 400x for audio with <6% Feature Error Rate loss, enabling battery-free operation on a commercial MCU via solar harvesting. We demonstrate practical impact in two real-world systems: an elk-detection camera trap (96.7% accuracy) and a phonetic-transcription wearable (8.44% FER) for child development research.

Zhihan Zhang, Alexander Le Metzger, Jiuyang Lyu +10

University of Washington · University of California San Diego · Northeastern University

Large Language Model-Guided OptimizationTiny MLHardware-Software Co-DesignEdge MLNeural Network CompressionOn-Device Inference
AskPDF
Jun 1, 2026

Toward a Modular Architecture for Embedded AI Agent Systems at the Edge

The rise of Large Language Models (LLMs) has enabled agentic AI capable of complex reasoning and tool use; however, deploying such autonomy in pervasive computing environments remains challenging due to the strict memory and energy constraints of embedded microcontrollers. Existing frameworks typically assume server-class resources or continuous connectivity, leaving a gap for deeply embedded systems. This paper proposes a modular reference architecture for Embedded Agent Systems that bridges the divide between deterministic real-time control and agentic intelligence. We introduce a tiered design that decouples On-Device Agents - executing highly compressed neural networks and rule-based logic for low-latency, privacy-critical tasks - from Cloud-Augmented Agents that leverage Small Language Models (SLMs) for higher-level reasoning and planning. A key contribution is the integration of a cross-cutting Governance Layer, ensuring observability, policy enforcement, and safety across distributed fleets of autonomous devices. Rather than presenting purely empirical benchmarks, we analyze architectural design principles and trade-offs regarding latency, energy, and reliable execution in resource-constrained environments.

Marcus Rüb, Michael Gerhards

Foresthub.Ai Villingen, Germany · Deloitte Consulting DÜsseldorf, Germany

AI Agent SafetyAI Agent GovernanceEdge ML
AskPDF