cs.HCApr 13, 2026
SaveFrom Words to Widgets for Controllable LLM Generation
Organizations: Cornell University Ithaca, NY, USA · University of Illinois Urbana-Champaign Champaign, IL, USA · The University of Texas at Austin Austin, TX, USA · Loyola University Maryland Baltimore, MD, USA
Abstract
Natural language remains the predominant way people interact with large language models (LLMs). However, users often struggle to precisely express and control subjective preferences (e.g., tone, style, and emphasis) through prompting. We propose Malleable Prompting, a new interactive prompting technique for controllable LLM generation. It reifies preference expressions in natural language prompts into GUI widgets (e.g., sliders, dropdowns, and toggles) that users can directly configure to steer generation, while visualizing each control's influence on the output to support attribution and comparison across iterations. To enable this interaction, we introduce an LLM decoding algorithm that modulates the token probability distribution during generation based on preference expressions and their widget values. Through a user study, we show that Malleable Prompting helps participants achieve target preferences more precisely and is perceived as more controllable and transparent than natural language prompting alone.
Figures & tables
Figure 1. Malleable Prompting . Instead of repeatedly refining outputs through underspecified natural-language prompts (left), Malleable Prompting lets users reify prompt preferences as GUI widgets and directly manipulate them to steer generation (A). Hovering over a widget reveals the output spans influenced by that preference, making individual effects inspectable (B). A two-part teaser figure contrasts natural-language prompting with malleable prompting. On the left, titled
Natural Language Prompting,'' a sequence of chat messages shows a user repeatedly revising a request for a funny blog post comparing the iPhone 15 and Pixel 8: asking to make it funnier, mention the camera, reduce humor, compare aesthetics, add sarcasm, add a table, and then revert an overly sarcastic result. Two illustrations identify the resulting interaction problems. The Gulf of Execution'' shows a user thinking through several nuanced and partially conflicting preferences, such as focusing on hardware, being funnier but less formal, and adding only a limited amount of sarcasm; the caption states that these preferences are hard to express and control. The Gulf of Evaluation'' shows the user looking uncertainly at a document version; the caption states that it is hard to understand how each preference affects the output. On the right, titled Malleable Prompting,'' the figure presents Malleable prompting as combining GUI-widget controllability with the flexibility of natural-language adjustments. A prompt contains visually emphasized preference phrases, including funny,'' 3 paragraphs,'' price,'' camera systems,'' and Add a table.'' A contextual widget panel converts these phrases into appropriate controls, including a scale slider, on/off toggle, pick-one selector, and step-value control. A blue annotation states, Reify Preferences as GUI Widgets to Steer Generation.'' Below, the generated blog-post text contains colored underlines and highlights corresponding to different preference widgets. An orange annotation states, ``Reveal How Each GUI Widget Influence the Output,'' indicating that hovering over a widget exposes the text spans it affects.| System | Domain | Steering | Attribution |
|---|---|---|---|
| Malleable Prompting | Text | Training-free; arbitrary spans | Span-level |
| PromptCanvas ( Amin et al., 2025b ) | Text | ✗ | ✗ |
| DirectGPT ( Masson et al., 2024 ) | Text | ✗ | ✗ |
| Steerable Chatbots ( Bo et al., 2025 ) | Text | Pretrained steering vectors | ✗ |
| PromptCharm ( Wang et al., 2024 ) | Image | Database-defined modifiers | Attention |
| AdaptiveSliders ( Jain et al., 2025 ) | Image | Pretrained LoRA | ✗ |
Table 1. Comparison of Malleable Prompting with closely related systems that combine prompts with direct manipulation. Malleable Prompting supports training-free steering of arbitrary user-defined preference spans and reveals each widget’s influence through span-level attribution. The table compares six systems by domain, steering method, and attribution support. Our system is the only text system that supports training-free steering of arbitrary user-defined spans and provides span-level attribution. PromptCanvas and DirectGPT provide neither steering nor attribution. Steerable Chatbots uses pretrained steering vectors but provides no attribution. The two image systems, PromptCharm and AdaptiveSliders, use database-defined modifiers and pretrained LoRA, respectively; only PromptCharm provides attribution, using attention.
| Dimension | Description | Examples | Typical Attribute(s) | Sources |
|---|---|---|---|---|
| Format | How the output is visually, logically, or syntactically organized. | “as a poem,” “in JSON,” “use bullet points,” “two columns” | Categorical, Binary | ( Lee et al., 2024 ; Malaviya et al., 2025 ; Kim et al., 2025 ; Zhu et al., 2026 ; Zhou et al., 2025 ) |
| Tone | The emotional resonance, voice, or stylistic flavor of the writing. | “funny,” “professional,” “academic,” “enthusiastic” | Categorical, Continuous | ( Lee et al., 2024 ; Malaviya et al., 2025 ; Kim et al., 2025 ) |
| Audience | Who the text is tailored for, or the role the LLM should assume. | “explain to a 5-year-old,” “for domain experts,” “act as a lawyer” | Categorical, Binary | ( Lee et al., 2024 ; Malaviya et al., 2025 ; Zhu et al., 2026 ; Zhou et al., 2025 ) |
| Length | Constraints on the physical size or detail level of the output. | “short,” “under 200 words,” “3 paragraphs,” “concise” | Numeric, Continuous | ( Lee et al., 2024 ; Malaviya et al., 2025 ; Zhou et al., 2025 ) |
| Content | Specific rules about what information to include or exclude. | “no jargon,” “include references,” “focus on the impact” | Categorical, Binary | ( Malaviya et al., 2025 ; Zhou et al., 2025 ) |
Table 2. Taxonomy of the preference space. The taxonomy organizes the preference space based on common dimensions identified in prior literature ( Lee et al., 2024 ; Malaviya et al., 2025 ; Kim et al., 2025 ; Zhou et al., 2025 ; Zhu et al., 2026 ) , along with the typical attribute types through which each dimension manifests. The table has five columns: Preference Dimension, Description, Examples, Typical Attribute(s), and Sources. It organizes user preferences for LLM outputs into five dimensions. Format and Structure refers to how output is visually, logically, or syntactically organized, with examples such as
as a poem,'' in JSON,'' use bullet points,'' and two columns,'' and is associated with categorical and binary attributes. Tone and Style refers to emotional resonance, voice, or stylistic flavor, with examples such as funny,'' professional,'' academic,'' and enthusiastic,'' and is associated with categorical and continuous attributes. Audience and Persona refers to who the text is tailored for or what role the model should assume, with examples such as explain to a 5-year-old,'' for domain experts,'' and act as a lawyer,'' and is associated with categorical and binary attributes. Length and Conciseness refers to constraints on output size or detail level, with examples such as short,'' under 200 words,'' 3 paragraphs,'' and concise,'' and is associated with numeric and continuous attributes. Content Constraints refers to rules about what information to include or exclude, with examples such as no jargon,'' include references,'' and focus on the impact,'' and is associated with categorical and binary attributes. The final column cites prior literature supporting each dimension.Figure 2. Making prompts malleable. Users can optionally enrich an initial prompt with suggested preferences (A.1), automatically parse controllable attributes and convert them into widgets or manually bind arbitrary selected text to a widget (C.1). Users can remove widgets via “Remove Malleability” (C.2). For categorical attributes, the system also suggests alternative values for later iterations (C.3). Three stacked interface panels illustrate a workflow for transforming a plain prompt into an interactive, malleable prompt with GUI controls. In the top panel, the prompt reads,
Write a blog post to compare iPhone 15 and Pixel 8,'' with a button labeled Enrich'' on the right; label A marks this stage. A curved arrow points to the middle panel, where the prompt has been automatically expanded to Write a funny blog post in 3 paragraphs comparing the price and hardware of iPhone 15 and Pixel 8. Add a table for comparison.'' A button labeled Parse'' appears on the right, and label B marks this stage. Another arrow points to the bottom panel, where parts of the prompt have been converted into editable widgets: funny'' is paired with a slider, 3 paragraphs'' with a numeric stepper, price'' with a pick-one dropdown, Add a table'' with an on/off toggle, and hardware'' is selected for control assignment. A pop-up menu labeled Make it malleable by'' shows options such as Auto-parsing,'' Scale,'' On / Off,'' Pick one,'' and Step value''; label C highlights this control-selection menu. A second dropdown for the price'' term lists alternatives including Price,'' Camera,'' Battery,'' and Design,'' with ``Design'' highlighted; label D marks this pick-one menu. Hand cursors and arrows emphasize the sequential interaction flow from enrichment, to parsing, to direct widget assignment.Figure 3. Inspecting widget effects on generated text. Clicking a widget reveals its localized influence on the output: affected sentences are underlined, positively induced phrases are highlighted in green, and discouraged wording is highlighted in red. A text-generation interface showing a prompt that reads,
Write a funny blog post in 3 paragraphs comparing the design and camera systems of iPhone 15 and Pixel 8. Add a table for comparison.'' The words funny,'' 3 paragraphs,'' design,'' camera systems,'' and table'' are highlighted in purple, with controls such as a funny slider set high, a paragraph count selector set to 3, and a design tag. A hand cursor points to the humor slider, and a doodled arrow and circle connect the prompt to the generated paragraph below. The output begins with a humorous comparison of the two phones, and several phrases are highlighted and underlined to show how specific prompt settings influenced the generated text.Figure 4. Tracking prompt iterations with a node-link graph. Versions generated with the same prompts and widgets appear along the same horizontal branch. Adding new preferences or widgets creates a new vertical branch. Visual encodings map widget values to node color for categorical attributes, node fill for binary attributes, and node size for continuous or numerical attributes. A visualization interface showing how prompt attributes of each iterations are encoded visually. Three dropdown menus at the top read
Color by hardware,'' Fill by Add a table for comp…,'' and Size by funny.'' Below them, a node-link style chart displays colored circles connected by horizontal and stepped lines. Circle color represents hardware-related categories, with blue for hardware, green for cameras, and orange for design. Circle fill indicates whether the attribute Add a table for comparison'' is true or false, using filled versus hollow markers. Circle size represents the degree of ``funny'' on a scale from 0.00 to 3.00, with larger circles indicating higher values. Legends at the bottom explain the color, fill, and size mappings.| Measure | Task | Slider | Toggle | Dropdown | Stepper | Total |
|---|---|---|---|---|---|---|
| Created | Jeopardy | 181 (52.31%) | 86 (24.86%) | 54 (15.61%) | 25 (7.23%) | 346 |
| Exploration | 96 (52.17%) | 44 (23.91%) | 38 (20.65%) | 6 (3.26%) | 184 | |
| Total | 277 (52.26%) | 130 (24.53%) | 92 (17.36%) | 31 (5.85%) | 530 | |
| Adjusted | Jeopardy | 385 (83.70%) | 46 (10.00%) | 21 (4.57%) | 8 (1.74%) | 460 |
| Exploration | 274 (88.67%) | 17 (5.50%) | 10 (3.24%) | 8 (2.59%) | 309 | |
| Total | 659 (85.70%) | 63 (8.19%) | 31 (4.03%) | 16 (2.08%) | 769 |
Table 3. Distribution of widget types across creation and adjustment. This summary includes the raw counts and percentages of widget types that participants created and adjusted across both the Jeopardy evaluation and free-form exploration tasks. The table summarizes the distribution of widget types across creation and adjustment in the
Jeopardy'' and Exploration'' tasks. For created widgets, 530 total widgets were created: sliders were most common at 277 (52.26%), followed by toggles at 130 (24.53%), dropdowns at 92 (17.36%), and steppers at 31 (5.85%). For adjusted widgets, 769 total adjustments were made: sliders dominated at 659 (85.70%), followed by toggles at 63 (8.19%), dropdowns at 31 (4.03%), and steppers at 16 (2.08%). This shows that sliders were by far the most frequently created and especially the most frequently adjusted widget type across both tasks.| Task | Type | Format | Tone | Audience | Length | Content | Total |
| Jeopardy | Slider | 30 (16.6%) | 89 (49.2%) | 2 (1.1%) | 30 (16.6%) | 30 (16.6%) | 181 |
| Toggle | 28 (32.6%) | 2 (2.3%) | 0 (0.0%) | 1 (1.2%) | 55 (64.0%) | 86 | |
| Dropdown | 0 (0.0%) | 4 (7.4%) | 26 (48.1%) | 1 (1.9%) | 23 (42.6%) | 54 | |
| Stepper | 9 (36.0%) | 0 (0.0%) | 0 (0.0%) | 15 (60.0%) | 1 (4.0%) | 25 | |
| Subtotal | 67 (19.4%) | 95 (27.5%) | 28 (8.1%) | 47 (13.6%) | 109 (31.5%) | 346 | |
| Exploration | Slider | 16 (16.7%) | 54 (56.2%) | 0 (0.0%) | 23 (24.0%) | 3 (3.1%) | 96 |
Table 4. Distribution of widget types across preference categories. This summary details the frequency and categorization of widget types across preference categories (Table 2 ) created by participants during both the Jeopardy evaluation and free-form exploration tasks. The table summarizes the distribution of created widget types across five preference categories in the
Jeopardy'' and Exploration'' tasks. Across both tasks, 530 widgets were created. Tone and Style was the most common category at 160 widgets (30.2%), followed by Content Constraints at 149 (28.1%), Format and Structure at 102 (19.2%), Length and Conciseness at 72 (13.6%), and Audience and Persona at 47 (8.9%). Sliders were used mainly for Tone and Style, toggles for Content Constraints, dropdowns for Audience and Persona, and steppers most often for Length and Conciseness. This pattern appeared in both tasks, though Exploration had a somewhat larger share of Tone and Style widgets than Jeopardy.Figure 5. Perceived support for design goals. Participants’ responses to a 7-point Likert-scale questionnaire, comparing our system against the baseline condition across both execution-related (Q1–Q4) and evaluation-related (Q5–Q6) measures.
Malleable Prompting'' versus Baseline'' stacked horizontal Likert chart for six statements about whether the system helps users express and refine output preferences, rated from 1 (strongly disagree) to 7 (strongly agree). Malleable Prompting'' is strongly positive across all six questions, with responses concentrated in ratings 5 to 7. For Q1, all 12 responses are positive (3 slightly agree, 6 agree, 3 strongly agree). Q4 and Q6 are also entirely positive, with Q4 showing 7 slightly agree and 5 strongly agree, and Q6 showing 5 slightly agree and 7 strongly agree. Q3 and Q5 each include only one slightly disagree and one neutral response, with the rest positive. In contrast, the Baseline'' condition is much more mixed and often negative or neutral. Q1 and Q2 show some agreement but also include disagreement and neutral responses. Q3 through Q6 are dominated by ratings 1 to 4, with only a few agreement responses. The largest gaps favoring ``Malleable Prompting'' appear for discovering new preferences, iterating closer to the desired output, attributing output changes to inputs, and comparing or restoring versions.Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Attribute | Example output excerpt | |
|---|---|---|
| enthusiastic | 0 | Join us for Campus Sustainability Day on April 22nd. The event will include workshops, activities, and opportunities to learn about eco-friendly practices and environmental responsibility. Attend to learn how you can support a more sustainable campus community. |
| 1 | Join us for Campus Sustainability Day on April 22nd! The event will feature workshops, activities, and opportunities to learn about eco-friendly practices and environmental responsibility. Come be part of a more sustainable campus community. | |
| 2 | Get ready to make a difference at Campus Sustainability Day on April 22nd! Join us for engaging workshops, hands-on activities, and inspiring opportunities to learn about eco-friendly practices and environmental responsibility. Come celebrate and help build a more sustainable campus community! | |
| 3 | Get ready to celebrate, connect, and make a difference at Campus Sustainability Day on April 22nd! Join us for exciting workshops, hands-on activities, and inspiring opportunities to explore eco-friendly practices and environmental responsibility. Let’s come together and show our love for a greener future by building a more sustainable campus community! | |
| playful | 0 | A large language model is an advanced type of artificial intelligence that is trained on vast amounts of text data to understand and generate human-like text. It can perform tasks such as answering questions, writing stories, coding, and translating languages, but it cannot truly understand or possess consciousness. Additionally, it may sometimes produce incorrect or biased information, so its outputs should be critically evaluated. |
| 1 | A large language model is like a super-smart digital brain that’s been trained on tons of text to understand and generate human-like language—think of it as a chatbot with a PhD ! It can write stories, answer questions, translate languages, and even help with coding, but it can’t truly understand emotions or have real-world experiences like humans do. Basically, it’s a helpful tool with a lot of knowledge, but it’s still just a bunch of words and patterns—it doesn’t have a soul, just a lot of data! |
Table 5. Example effects of across continuous attributes. Outputs produced from the same prompts while varying the strength of target attributes during decoding. The top example shows the attribute enthusiastic for prompt “ Write a short post announcing a campus sustainability event in an enthusiastic tone ,” and the bottom example shows playful for prompt “ Explain what a large language model is and what it can/cannot do in a playful way. ” In both cases, larger values produce a stronger realization of the target attribute while preserving the underlying content. Highlighted spans indicate wording influenced by the target attribute. The table compares the effect of increasing decoding strength across two continuous attributes,
enthusiastic'' and playful.'' The first column lists the attribute, the second lists values 0 through 3, and the third shows example output excerpts. For enthusiastic, the prompt is about announcing Campus Sustainability Day. At , the announcement is neutral and informational. As increases to 1, 2, and 3, the same event description becomes progressively more energetic, with highlighted phrases such as Join us,'' Get ready to make a difference,'' and Let's come together and show our love for a greener future.'' For *playful*, the prompt asks for an explanation of what a large language model is and what it can or cannot do. At $\lambda=0$, the explanation is formal and straightforward. As $\lambda$ increases, the explanation becomes progressively more whimsical and exaggerated, using highlighted phrases such as super-smart digital brain,'' super-smart, never-sleeping librarian,'' and super-smart gossip squad.'' Overall, the table shows that larger values produce a stronger realization of the target attribute while preserving the underlying content of each prompt.| Scope | Baseline [95% CI] | Ours [95% CI] | [95% CI] | p |
|---|---|---|---|---|
| Overall | 0.306 [0.278, 0.335] | 0.576 [0.546, 0.607] | +0.270 [0.228, 0.312] | |
| 0.450 [0.382, 0.518] | 0.741 [0.673, 0.809] | +0.291 [0.195, 0.388] | ||
| 0.306 [0.255, 0.357] | 0.711 [0.661, 0.760] | +0.404 [0.333, 0.476] | ||
| 0.259 [0.220, 0.298] | 0.432 [0.388, 0.476] | +0.173 [0.115, 0.232] |
Table 6. Slider-responsiveness regression. Least-squares slopes of judge score on intensity level (1-7). is measured in judge-score points per slider step. The table reports least-squares slopes relating intensity level to judge score for the baseline and our method. Overall, our method has a substantially larger slope than the baseline, 0.576 versus 0.306, with a difference of 0.270. This improvement is significant across all conditions, with the largest difference at (k=2), followed by (k=1) and (k=3). All comparisons are statistically significant at (p<.001), indicating that our method responds more strongly and consistently to changes in intensity level.
| Condition | Sign-agreement [95% CI] | |
| H1: Faithfulness | ||
| Highlighted | 81.4% [81.0, 81.7] | 598 |
| Neutral (control) | 67.2% [66.4, 68.0] | 598 |
| by label polarity | ||
| Positive | 85.1% [84.8, 85.4] | 598 |
| Negative | 58.1% [57.0, 59.2] | 580 |
Table 7. Attribution faithfulness and specificity. Directional sign-agreement between a token’s displayed attribution label and the observed direction of its probability change under teacher forcing. Higher is more faithful. The neutral band (H1) and off-diagonal (H2) rows serve as controls that should sit near chance. is the number of prompt-attribute settings aggregated. The table reports directional sign-agreement for attribution faithfulness and specificity. Highlighted tokens achieve 81.4% agreement, compared with 67.2% for neutral tokens. Positive attribution labels are more faithful than negative labels, with 85.1% versus 58.1% agreement. Agreement remains near 81% across settings with one, two, or three active attributes. For specificity, agreement is 81.0% when adjusting a token’s own attributed preference, compared with 60.8% when adjusting another preference, indicating that the attribution signal is widget-specific.
| ID | Gender | Age | Writing* | Prompting** | Writing Type in Free-Form Exploration |
|---|---|---|---|---|---|
| P01 | Male | 29 | 1–3 hours | 5 | Research article |
| P02 | Female | 24 | 8–14 hours | 5 | Homework essay and slide script |
| P03 | Female | 25 | 15+ hours | 5 | Activity announcement |
| P04 | Male | 24 | 4–7 hours | 6 | Argumentative essay |
| P05 | Female | 26 | 4–7 hours | 5 | Social media post |
| P06 | Female | 27 | 4–7 hours | 5 |
Table 8. Participant information. This summary presents participants’ demographic information and the self-chosen writing task they explored in the free-form exploration task in our user study. *Writing volume in a typical week; **Prompting experience was self-reported on a 7-point scale, where 1 = Not at all experienced and 7 = Very experienced. The table summarizes participant demographics and the self-chosen writing task used in the free-form exploration of the user study. The columns are participant ID, gender, age, weekly writing volume, self-reported prompting experience on a 7-point scale, and writing type selected for free-form exploration. The study includes 12 participants, labeled P01 through P12. Ages range from 24 to 37. There are 5 male participants and 7 female participants. Weekly writing volume ranges from 1–3 hours to 15+ hours, with most participants reporting 4–7 hours. Prompting experience ranges from 4 to 6, with most participants reporting 5. The writing tasks selected for free-form exploration are diverse and include research article, homework essay and slide script, activity announcement, argumentative essay, social media post, email, blog post, poem, reflection after watching a musical, and meeting announcement. Research article and email each appear twice, while the other writing types appear once. Overall, the table provides an overview of participant background and the range of writing contexts represented in the study.
Figure 6. An example screenshot of the baseline interface An
AI Chatbot'' interface showing an iterative writing interaction. The interface has a large chat panel with the title AI Chatbot'' in the upper left and an information icon in the upper right. In the conversation, the user first enters a request asking the chatbot to write an email to a team colleague about an immediate schedule change, explaining that a project deliverable deadline has been moved to next Friday because of unexpected API integration delays. The chatbot responds with a draft email including the subject line Urgent: Project Deliverable Deadline Updated to Next Friday,'' followed by a professional message explaining the delay, the revised deadline, and a request to adjust schedules accordingly. Below that, the user sends a short follow-up message, Make it more formal.'' The chatbot then generates a revised version of the email with more formal phrasing, including the updated subject line Urgent: Revised Deadline for Project Deliverable – Updated to Next Friday.'' At the bottom of the interface is a text input box labeled Ask me anything…'' with a send button. The figure illustrates iterative refinement of writing through natural-language feedback in a chat-based interface.| Task | Scenario | Version 1 | Version 2 | Version 3 |
|---|---|---|---|---|
| Write an email about an immediate schedule change for a project deliverable, adapting it to different recipients. | Recipient: a team colleague Formality: informal Tone: informative Format: short paragraphs | Recipient: the manager Formality: formal Tone: persuasive Format: bullet points | Recipient: the client Formality: formal Tone: apologetic Format: short paragraphs | |
| Explanation | Explain what a large language model is and what it can/cannot do, adapting the explanation to different audiences. | Audience: tech-savvy users Tone: professional Conciseness: detailed & technical Format: bullet points | Audience: general public Tone: approachable Conciseness: neutral Format: short paragraphs | Audience: middle-schoolers Tone: playful & encouraging Conciseness: simple & concise Format: short paragraphs |
Table 9. Jeopardy evaluation tasks and target preferences. Each task includes three target versions that differ along preference dimensions (e.g., audience, tone, formality, format, conciseness) based on the taxonomy in Table 2 and prior work ( Lee et al., 2024 ) , while keeping task facts (e.g., dates and key details) constant across versions. Supplemental materials include all target outputs presented to participants. The table summarizes Jeopardy evaluation tasks and preferences of target outputs. It contains two tasks,
Email'' and Explanation,'' and three target versions for each. For the email task, the versions vary by recipient, formality, tone, and format: colleague with informal informative short paragraphs, manager with formal persuasive bullet points, and client with formal apologetic short paragraphs. For the explanation task, the versions vary by audience, tone, conciseness, and format: tech-savvy users with a professional detailed bullet-point explanation, general public with an approachable neutral explanation in short paragraphs, and middle-schoolers with a playful, encouraging, simple explanation in short paragraphs.More in this topic
LLM Prompting
Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.
See all 429 papers