Prompt Less, Smile More: Semantic Engineering in Lieu of Prompt Engineering
Organizations: University of Michigan, Ann Arbor, USA
Abstract
AI-integrated applications increasingly rely on large language models (LLMs), but their intended behavior is not always fully specified by code. Meaning Typed Programming (MTP) automates prompt construction from program structure, yet selection criteria, output requirements, and application policies can remain unstated. We introduce Semantic Engineering, a methodology for identifying such missing intent and selectively adding and refining model guidance. We realize this methodology through Semantic Context Annotations (SemTexts), a dedicated language mechanism independent of documentation and schema metadata. SemTexts associate natural-language instructions with individual program entities and use declaration scopes to determine where they apply. Our implementation extends the Jac compiler and MTP runtime to preserve these associations and select applicable guidance during automatic prompt construction. We evaluate Semantic Engineering on seven AI-integrated applications and two standard benchmarks across three models. Across the applications, it achieves a 5.46-fold reduction in integration lines of code relative to Prompt Engineering while delivering equal or better task performance in most model-application settings, reaching up to 1.83 times the task performance of Prompt Engineering.
Figures & tables
| Benchmark Application | Task Description and Dataset | Evaluation Method | Capabilities |
| Memory Retrieval | Retrieve relevant memories from a user’s personal memory database to answer a query. Dataset: 20 user profiles with 10 memories and five questions per user; 100 question instances, including 95 unique instances. | Macro-averaged F1, averaged over three runs. | C3, C4 |
| Image Extraction | Extract structured information and describe visual features from images. Dataset: A fixed manifest of 300 images sampled from LAION-400M ( Schuhmann et al., 2021 ) . | Hybrid score combining semantic similarity through cosine similarity and lexical matching through BM25. | C1, C4 |
| Task Manager | Manage task creation and completion, task summarization, and email drafting and sending. Dataset: 220 synthetic queries comprising 100 task-handling, 20 task-summary, and 100 email-handling requests. | LLM-judged success rate using a fixed rubric that checks routing, request-specific validity, and applicable date calculations. | C2, C3, C4, C5 |
| Content Creator | Perform writing tasks through a multi-agent workflow with planning, writing, and reviewing stages. Dataset: 200 InstructEval prompts ( Chia et al., 2023 ) across four writing categories: informative, professional, argumentative, and creative. | LLM-judged success rate. A response is successful when it satisfies at least 11 of 15 evaluation criteria. | C3, C4, C5, C6 |
| Aider Genius | Resolve software issues through a planner that guides the Aider code editor in modifying an existing repository. Dataset: 60 SWE-bench Lite instances ( Jimenez et al., 2024 ) . | Percentage of instances resolved under the SWE-bench test-based evaluation. | C2, C3, C5, C6 |
| Service Center | Resolve customer-service tickets using a router, seven specialist agents with distinct toolsets, and a reply merger. Specialists handle refunds, cancellations, status checks, credentials, email, rescheduling, and escalation. Dataset: A frozen set of 240 synthetic ticket episodes executed against a mock backend. | End-to-end pass rate, determined by comparing the resulting backend state with the expected canonical state, following -bench ( Yao et al., 2024 ) . | C2, C5, C6 |
LLM Prompting
Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.
See all 450 papers