Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often has a richer structure and semantics, which is not conveyed in a string. For example, most prompts contain both instructions ("Summarize this paper into a paragraph") and data (the paper to summarize), but these are usually not distinguished when passed to the model. This can lead to model confusion and security risks, such as prompt injection attacks. This work addresses this shortcoming by introducing an LLM-native mark-up language, LLMON (LLM Object Notation, pronounced "Lemon"), that enables the structure and semantic metadata of the text to be communicated in a natural way to an LLM. This information can then be used during model training, model prompting, and inference implementation, leading to improvements in model accuracy, safety, and security. This is analogous to how programming language types can be used for many purposes, such as static checking, code generation, dynamic checking, and IDE highlighting. We discuss the general design requirements of an LLM-native markup language, introduce the LLMON markup language and show how it meets these design requirements, describe how the information contained in a LLMON artifact can benefit model training and inference implementation, and provide some preliminary empirical evidence of its value for both of these use cases. We also discuss broader issues and research opportunities that are enabled with an LLM-native approach.
Figures & tables
Attribute
LLMs
Programmed Systems
Instruction/Data Separation
No
Yes, supported by high-level languages and operating systems
Referential Structure
No
Yes, parameters, variables, etc.
Execution Control
No
Yes, semantics of execution are well defined
Program Portability
No
Yes, via compilers or lang VMs
Deterministic Execution
No
Yes, in most cases
Nonlinear data flow
No
Yes, intermediate assignments to unrelated variables are independent
Table 1: Comparison Between Programming via Prompts with LLMs and Traditional Programming
Design Requirement
LLMON Approach
1. User-defined Tags
supported with \tagname\text /tagname/
2. Anchors for Instances
supported with “:” special token
3. Explicit Nesting Names
supported with “.” special token
4. Prefix
uses prefix-style tags
5. Convertibility
have converters between JSON and LLMON
6. No Escaping
LLMON does not use any escaping
Table 2: LLMON Design Decisions
LLMON:
LLMON_ITEM LLMON_LIST
LLMON_LIST:
LLMON_ITEM LLMON_LIST
| ϵ
LLMON_ITEM:
USER_TYPE | | OBJECT | LIST
| integer | float | string
Figure 5: LLMON Grammar. terms in UPPER_CASE are nonterminals. terms in single quotes or lowercase are terminals. The table provides current values for the nonquoted terminals. “ user_tag_text ” represents a typical identifier extended to include “ : ” and “ . ” as illustrated in examples in Section 3.1 . “ object ”, “ item ”, and “ list ” are exactly those strings.
LLMON Special Tokens
<|open|>
<|open_end|>
<|close|>
<|self_close|>
<|.|>
<|:|>
Table 3: LLMON Special Tokens and Syntax Mapping from Shorthand
Terminal
String
start_user_tag
<|open|> user_tag_text <|close|>
end_user_tag
<|open_end|> user_tag_text <|close|>
self_close_user_tag
<|open|> user_tag_text <|self_close|>
start_object_tag
<|open|>object<|close|>
end_object_tag
<|open_end|>object<|close|>
start_object_item_tag
<|open|>item<|close|>
Figure 6: Current values for the nonquoted terminals from Figure 5 expressed in LLMON. “ user_tag_text ” is the same as in Figure 5 . It can be any identifier and can include “ : ” and “ . ” as illustrated in examples in Section 3.1 . “ object ”, “ item ”, and “ list ” are exactly those strings.
Figure 9: Workflow from Shorthand syntax to the token sequence used during model post-training and inference. A structured artifact written in Shorthand syntax is converted into LLMON, where structural delimiters are represented using special tokens (shown in red). The tokenizer then maps the sequence into token identifiers, preserving structural boundaries in the token stream. Blue and black tokens represent ordinary text tokens produced by the tokenizer; alternating colors are used only to visually distinguish adjacent tokens and illustrate token boundaries as seen by the tokenizer. This tokenized representation can be used directly for model post-training and inference procedures.
Model
Tokens
MMLU
GSM8K
IFEval
Distractor
Base (no post-training)
Granite-4.0-Micro-Base
–
61.52
0.00
38.78
4.40
Qwen-2.5-3B
–
65.09
0.15
27.13
27.80
Full fine-tuning baseline (post-training with chat-template data)
Granite-4.0-Micro-Base
2.9B
39.95
0.61
72.07
0.00
Qwen-2.5-3B
2.9B
48.51
1.13
74.05
0.40
Table 4: Comparison of base models, conventional post-training baselines (chat-template instruction tuning), and LLMON-structured post-training. MMLU, GSM8K, and IFEval are reported in zero-shot accuracy, and “Distractor” denotes accuracy on our structured multi-instruction Distractor benchmark. Token counts indicate the amount of post-training data used.
Model
Method
Distractor
Qwen2.5-3B-Instruct
baseline
43.20
constraint mask with LLMON
69.00
Granite-3.3-8B-Instruct
baseline
41.60
constraint mask with LLMON
72.00
Granite-4.0-Micro
baseline
40.80
constraint mask with LLMON
72.40
Table 5: Comparison of models applying constraint masking with LLMON at inference on multi-instruction Distractor benchmark.
Structured prompting organizes instructions, context, reasoning steps, or output schemas to make language-model behavior more reliable and controllable.