cs.HCSep 21, 2026
SaveGenerative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks
Organizations: University of Michigan Ann Arbor, MI, USA
Abstract
Visual instructions for physical tasks are typically authored in one context and followed in another, requiring users to translate demonstrated tools, materials, and spatial relationships into their own environment. We introduce Generative Tutorial, a conceptual framework for live visual instruction that depicts intended outcomes and actions within the user's environment and task flow. A formative evaluation of state-of-the-art image and video generation identifies failures and potential benefits across 15 physical tasks. Drawing on these findings, we build an augmented-reality prototype system that proactively generates goal images and demonstration videos using observed workspace context and predicted visual outcomes of preceding actions. A 24-participant lab study found higher task performance quality, greater perceived workspace correspondence, and shorter step-confirmation intervals with the system than with pre-authored guidance. Qualitative findings highlighted how contextual resemblance shapes trust, how generation errors affect interpretation, and how guidance delivery should adapt to users' needs, informing future designs.
Figures & tables
Figure 1. Generative Tutorial reimagines tutorials as live, contextualized visual instructions adapted to a user’s environment and evolving task progress. The prototype system continuously captures real-world context and grounds the task model in the user’s environment. For each step, it aims to generate a workspace-grounded goal image and demonstration video (e.g., showing how to press and twist a muddler to crush ice). By propagating real and speculative visual context forward, the system proactively generates guidance for upcoming steps (e.g., squeezing juice and preparing a mint garnish). The generated guidance is presented and situated in its associated task workspace, allowing instructions to coexist across activities and remain available as the user switches between them (e.g., moving from preparing a house mocktail to setting the dining table). Composite overview of Generative Tutorial's contextualized visual instruction capabilities and system workflow. The left section shows the environmental and temporal misalignment inherent in static visual tutorials, paired with an illustration of a user struggling to map a floating 2D video to their physical environment. The center section shows task model contextualization for a user wearing a mixed-reality headset at a kitchen counter, utilizing real and speculative context to dynamically generate workspace-grounded goal images and demonstration videos for their immediate task. The right section shows proactive generation for future dependent steps, such as squeezing juice and preparing a mint garnish, alongside workspace distribution scenarios where generated instructions remain spatially situated as the user switches activities to set the dining table.
Figure 2. Conceptual framework for Generative Tutorial , illustrated through flower arranging. Environment Modeling represents observed entities, states, and relationships as visual context. Action Grounding relates an action in the task model, including its objects and preconditions, to the current scene. Once sufficient context is available, Visual Instruction Generation depicts how to act or what the action should achieve. Environment alignment connects the depiction to the physical scene, while timeliness concerns its availability at the time of need. User interpretation and action lead to changes in the environment and task state that inform subsequent guidance. Composite overview of Generative Tutorial's conceptual framework, illustrated through a flower arranging task. The Environment Modeling section shows the system observing the current physical state to extract visual context, including entities, states, and relationships. The Action Grounding section shows a task model relating a specific action, such as inserting a flower, along with its involved objects and preconditions, to the current scene. The Visual Instruction Generation section shows the creation of a visual instruction once context is ready, paired with an environment alignment step that maps the depiction to the physical scene, and a timeline demonstrating that the system's latency must satisfy the user's timeliness need before the user interprets, acts, and updates the environment state.
Figure 3. Formative model evaluation flow. A unit-action instruction and initial reference frame condition image and video generation across two model snapshots. Two researchers examine each returned artifact against both inputs and assign observable issue codes. In the illustrated example, the generated image uses the matcha glass rather than the milk bowl as the receiving vessel (A4: Object or subpart identification error), placing the combined contents in the glass while leaving the milk bowl empty (B8: Incorrect spatial relationship); an additional whisk appears alongside the original, indicating key-object duplication (B9: Key-object integrity failure). The video receives no issue codes. Composite overview of the formative model evaluation flow across four processing stages. Stage 1 shows the probe input, consisting of a unit-action text prompt for mixing matcha paired with an initial reference starting scene image. Stage 2 and Stage 3 show the generation and returned media phases, contrasting image models like Gemini 3.1 Pro and Flash against video models like Veo 3.1 and Gemini Omni Flash, complete with median generation latencies and representative output artifacts. Stage 4 shows the researcher labeling process, detailing specific visual error codes assigned to the generated image—such as object identification, spatial relationship, and duplication errors—while assigning no issues to the video, all scoped to an evaluation formula of sampled steps and tasks.
Figure 4. Representative failures in generated guidance. Each panel pairs a condensed instruction with its reference frame and generated output. (a) The black and gray LEGO pieces are joined end-to-end instead of stacked (B1). (b) The mixture appears in the wrong vessel (A4/B8), and a second whisk appears while the original remains (B9). (c) The yoga video adds body and foot adjustments beyond the requested posture (B4). Orange outlines highlight relevant image regions. Composite overview of representative failures in generated visual guidance across different task models. Panel (a) shows an 'Incorrect attachment' failure in a LEGO assembly task, paired with reference and generated images demonstrating that pieces were joined end-to-end rather than stacked as instructed (B1). Panel (b) shows 'Altered target and scene content' during a matcha preparation task, highlighting that the mixture appeared in the wrong vessel (A4/B8) alongside an erroneously duplicated whisk (B9). Panel (c) shows an 'Action overrun' in a yoga video sequence, demonstrating extra body and foot adjustments that continue beyond the initially requested tabletop posture (B4).
Figure 5. Prototype walkthrough illustrated through shelf organizing. (1) Set up the task: the user selects a task and captures the relevant workspaces. (2) Observe the current scene: as the user approaches the shelf, the headset captures its current arrangement to inform guidance generation. (3) Follow the guidance: the user compares the goal image and demonstration video with the physical scene and performs the action, here stacking dishes on the shelf. Composite overview of a prototype walkthrough illustrated through a shelf organizing task. Panel (1) shows a user setting up the task by selecting relevant workspaces in a kitchen environment, paired with bounding box overlays capturing the spatial layout. Panel (2) shows the system observing the current scene from a first-person perspective as the user approaches the shelf to capture its initial physical arrangement. Panel (3) shows the user following the guidance to physically stack dishes on the shelf, guided by floating, system-generated goal image and demonstration video overlays situated within the physical workspace.
Figure 6. Visual state propagation illustrated through shelf organizing. Orange denotes camera observations (real context), blue predicted outcomes (speculative context), and purple demonstration video. The initial observation contains two workspace-specific subcontexts: the shelf ( ) and the staging surface containing the tray ( ). (A) Sequential prediction uses each predicted outcome as the next action’s source. (B) Two-action prediction generates the combined outcome in one request, providing an alternative route to the same intended state. (C) Video generation uses source and predicted outcome images as start and end frames. (D) Regrounding replaces speculative context with a fresh observation. Red annotations mark observed errors, not automatic detection. Composite overview of the context-driven generation pipeline for goal images and demonstration videos. Flow (A) shows the generation of a workspace-grounded goal image for stacking dishes derived directly from real subcontext camera observations. Flow (B) shows speculative generation chains for future tasks like clearing a shelf or emptying a tray, paired with annotations highlighting hallucination failures such as extra gloves or a missing kettle. Flow (C) shows the generation of a demonstration video utilizing the initial observation as a start frame and the generated goal image as an end frame. Flow (D) shows a regrounding observation where the system captures the updated real context of the workspace to correct prior speculative errors and accurately generate the final goal image.
Figure 7. User-study apparatus, conditions, and tasks. Both conditions present guidance beside the physical workspace using the same layout; they differ in whether visual guidance is authored in advance or generated during task performance. Each participant completes all four tasks once, with two tasks per condition. Task order and task–condition assignments are counterbalanced across participants. Composite overview of the user-study apparatus, experimental conditions, and physical tasks. The left section shows the mixed-reality interface from a participant's perspective, demonstrating how both conditions present goal images, demonstration videos, and text instructions in an identical spatial layout beside the physical workspace. The top right section shows the two core visual instruction conditions—comparing manually pre-authored guidance against system-generated guidance. The bottom right section shows the physical material setups for the four evaluated tasks: Strawberry matcha latte, Red-bean mochi, Dinner Table setting, and Spiral bouquet, which are counterbalanced across the two conditions for each participant.
Figure 8. Task quality by task, step, and condition. Points show mean scores. Composite overview of task quality by task, step, and condition, detailing mean scores and 95% bootstrap confidence intervals. The top section presents dot-and-whisker plots for the Matcha latte and Mochi tasks, illustrating generally high step quality scores across both instruction types, with generated guidance frequently approaching or hitting maximum scores. The bottom section contrasts the Table setting and Bouquet tasks; the Table setting panel highlights a notable variance and quality dip for generated instructions specifically on the 'Place dessert utensils' step, while the Bouquet panel shows generated guidance achieving consistently near-perfect scores that outperform the pre-authored baseline.
Figure 9. Two paired examples of visual guidance at steps with large descriptive condition differences. Generated guidance was associated with higher criterion completion for foam (6/12 to 11/12); the close pre-authored table-setting frame was associated with higher spoon-and-fork placement completion (10/12 to 6/12). Composite overview of condition-specific goal-image comparisons across tasks. A two-row, three-column grid contrasts pre-authored source-tutorial goal frames with generated workspace-contextualized goal images. The Matcha foam column shows higher criterion completion under the generated condition, whereas the Table-setting utensil-placement column shows higher completion under the pre-authored condition.
Figure 10. Action timing by condition. (a) Mean durations before action, during action, and before confirmation, illustrated with Step 2 of the Matcha latte task: before action, the participant identifies the water pitcher; at confirmation, the participant check final state. (b) Adjusted confirmation interval differences (Generated minus Pre-authored), with 95% confidence intervals. * . Composite overview of action timing by condition. Panel (a) shows mean durations for the before-action, action, and confirmation phases. Inset photographs illustrate Step 2 of the Matcha latte task: before acting, the participant identifies the water pitcher, and at confirmation, the participant verifies the final state. Initial planning and action-execution times do not differ significantly between conditions, whereas the confirmation phase is significantly faster by 1.06 seconds with generated instructions. Panel (b) shows adjusted generated-minus-pre-authored confirmation-interval differences, with 95% confidence intervals, for Matcha latte, Mochi, Table setting, and Bouquet. The reductions are statistically significant for Mochi and Table setting.
Figure 11. Participant ratings by condition. Points show mean (a) agreement with questionnaire statements and (b) goal-image helpfulness by task. * p < .05. Composite overview of participant ratings by condition across questionnaire statements and task-specific helpfulness. Panel (a) shows mean agreement with various subjective metrics on a 1–7 scale, illustrating that the generated condition significantly improved perceived correspondence—specifically how well the guidance matched the workspace and was viewed together with the physical task—compared to the pre-authored baseline, while maintaining similar levels of confidence and ease of understanding. Panel (b) shows goal-image helpfulness on a 1–5 scale across four activities, demonstrating that while helpfulness ratings were relatively uniform between conditions for the matcha, mochi, and table setting tasks, the generated goal images provided a statistically significant benefit for the visually complex spiral bouquet task.
Figure 12. Generated Media readiness relative to step start. Curves show cumulative shares of matched image and video arrivals under observed proactive timing and hypothetical requests at step start, holding recorded latencies fixed. Composite overview of Generated Media readiness relative to step start, contrasting proactive and reactive generation models. The left panel plots the cumulative share of goal image arrivals, demonstrating that observed proactive timing ensures nearly all images are available at the exact moment a step begins (0s delay), whereas a hypothetical system requesting generation at the step start would incur a median latency of 31.1 seconds. The right panel plots video arrivals, illustrating that proactive timing delivers the majority of videos instantly at step start, effectively eliminating what would otherwise be a median delay of 71.6 seconds under a reactive generation baseline.
Figure 13. Generated-media readiness by task and modality. Bars show the share of planned steps in each readiness category; segment labels give counts, and n gives the total planned steps. Composite overview of generated-media readiness broken down by task and modality. The top section covers the Matcha latte and Mochi tasks, illustrating robust proactive system performance where the vast majority of both generated images and videos were successfully 'Ready at start.' The bottom section covers the Table setting and Bouquet tasks, highlighting a significant reliability divergence between modalities; while goal images remained predominantly ready at the start of a step, demonstration videos frequently failed to process, represented by massive portions of the data falling into the 'No matching readiness log' category.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Task / source | Procedure scope | Steps | Motor involvement | Temporal organization | Object deformability | Environmental predictability |
|---|---|---|---|---|---|---|---|
| T1 | LEGO | Assemble a camera through stud alignment and attachment | 25 | Fine | Serial | Rigid | Closed |
| T2 | Crochet | Manipulate yarn loops into a slip knot and chain stitches | 10 | Fine | Serial | Deformable | Closed |
| T3 | Matcha latte | Measure, transfer, and mix drink ingredients | 7 | Mixed | Serial; continuous actions | Mixed | Closed |
| T4 | Origami | Crease and fold layered paper into a crane | 45 | Fine | Serial | Deformable | Closed |
| T5 | Furniture assembly | Assemble a chair using fasteners and mating frame joints | 28 | Mixed | Serial | Mixed | Closed |
| T6 | Gift wrapping | Measure, fold, and secure paper around a rigid box | 24 | Mixed | Serial | Mixed | Closed |
Table 1. Task metadata. The four-dimensional descriptive annotations are inspired by the literature on motor skills ( Muratori et al., 2013 ) and material distinctions ( Arriola-Rios et al., 2020 ) .
| Codebook | Explanation | Visual example |
|---|---|---|
| A. Interpretation and grounding | ||
| A1 Semantic overreach or intrusion E1, E3 | An unsupported object, person, part, or semantic element appears. Example: LEGO, step 4 (Gemini 3.1 Flash Image). The clip brick appears on an additional black assembly rather than only modifying the existing one. | Source Generated |
| A2 Ambiguity-resolution failure E1, E2 | The artifact adopts an unsupported resolution of ambiguity in the instruction or scene. Example: Packing a box, step 7 (Gemini 3.1 Flash Image). The tape placement resolves an underspecified sealing instruction in a way coded as unsupported. | Source Generated |
| A3 Hidden-state inference error E1, E3 | An occluded or latent property, such as orientation, connectivity, or a hidden surface, is inferred incorrectly. Example: Furniture assembly, step 6 (Gemini 3.1 Flash Image). The hidden fastening arrangement is inferred incorrectly while the bolt is being secured. | Source Generated |
| A4 Object or subpart identification error E1, E2, E3 | The artifact acts on, replaces, or emphasizes the wrong existing object or subpart. Example: Matcha latte, step 7 (Gemini 3.1 Flash Image). The green mixture and ice appear in the glass while the intended milk bowl is emptied. | Source Generated |
| B. Action and target-state accuracy | ||
Table 2. Formative evaluation codebook with source–output image pairs and sampled video frames.
| Task | Materials and tools | Prescribed steps |
|---|---|---|
| Strawberry matcha latte 7 steps | Matcha powder, water, milk, strawberries, ice, sieve, bowl, bamboo whisk, two spoons, and tall serving glass. | 0. Gather the ingredients and tools. 1. Pour about a teaspoon of matcha from the bag into the sieve and sift into the bowl. 2. Add a little water. 3. Whisk side to side until evenly foamed. 4. Reserve a garnish berry; remove the other berries’ tops and mash them in the glass. 5. Add ice and milk to three-quarters full, keeping the layers separate. 6. Pour matcha over a spoon to form a green layer; garnish and dust with matcha. |
| Red-bean mochi 7 steps | Red-bean paste, cornstarch in its yellow box, one prepared cling-film square, fully cooked mochi dough in a bowl, scraper or knife, filling tray, and serving plate. | 0. Gather materials; wait for the experimenter to confirm the dough is safe to handle. 1. Roll one red-bean filling ball. 2. Wrap it in the prepared cling film. 3. Dust the dough and surface with cornstarch; stretch the dough into a cylinder. 4. Divide into four equal portions and roll four dough balls. 5. Flatten one dough ball into a wrapper. 6. Unwrap the filling and set the film aside; center the filling, enclose it, and pinch and fold the edge inward. |
| Western-style table setting 5 steps | Twelve pieces: charger, dinner plate, bread plate, three forks (dinner, salad, dessert), two long knives, butter spreader, two spoons (soup, dessert), and drinking glass. Staging surface and dining table. | 0. Check all twelve pieces at the staging surface. 1. Place the charger at the table’s near edge. 2. Place forks on the left, knives and soup spoon on the right; align handles and turn blades inward. 3. Place dessert spoon and fork beyond the charger, with handles right and left, respectively. 4. Add the dinner plate, bread plate with spreader, and glass, preserving earlier placements. |
| Spiral bouquet 7 steps | Four artificial stems: one dark foliage stem, two cream/white roses, and one pink focal stem; one elastic loop. | 0. Check the four stems and elastic. 1. Hold the foliage stem in the left hand. 2. Cross one cream/white rose over it to form an X. 3. Add the second rose on the opposite side at the same crossing angle. 4. Turn the held bundle a quarter turn. 5. Cross the pink stem with the bundle; do not turn again. 6. Hook, wrap, and anchor the elastic around all four stems. |
Table 3. User-study tasks, materials, and step sequences. Step 0 is the material check; action steps begin at 1. Task names link to the YouTube sources used for pre-authored visual guidance.