cs.MMFeb 17, 2026

Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU

Authors: Rehana Mahfuz, Yinyi Guo, Erik Visser, Phanidhar Chinchili

Organizations: Qualcomm Technologies, Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Real-time conversational assistants for procedural manual tasks often depend on video input, which can be computationally expensive and compromise user privacy. For the first time, we propose a real-time conversational assistant that provides comprehensive guidance for procedural manual tasks using only lightweight privacy-preserving modalities such as audio and IMU inputs from a user's wearable device to understand the context. Using a furniture assembly task and a cooking task, we show how this assistant proactively communicates step-by-step instructions to a user performing a procedural task, and answers user questions. We illustrate the data generation method and the system design to achieve such an assistant. On observing that an off-the-shelf language model is a talkative assistant but is not always able to answer questions correctly, we demonstrate how finetuning the model improves its ability to limit unnecessary dialogues with a 50% increase in the precision, while also improving its ability to answer questions correctly, measured by a 150% increase in the recall of answers. We further describe how such an assistant is implemented on an edge device with no dependence on the cloud.

Figures & tables

Figure 1: Design of our proactive situated conversational assistant. Occurrences of user comments or recognized activities trigger calls to the language model that provides responses as necessary.
Figure 2: Sequential steps for the procedural tasks.
Figure 3: Human evaluation of the dataset quality.
Figure 4: Activity recognition classification result.
Figure 5: Performances of different language models as assistants for both tasks.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

You are an impartial judge who is evaluating the quality of responses provided by an AI assistant to a user who is <assembling a table/making soup>.
Here are the instructions to <assemble a table/make soup>:
<instructions including materials, steps and notes>
First you will be given a reference conversation where the assistant’s responses are exemplary. Then you will be given the conversation that occurred with the actual assistant. Your task is to evaluate the quality of the actual assistant’s responses based on correctness, helpfulness and restraint. Restraint refers to the assistant’s ability to avoid saying something unless necessary.
Please provide one overall rating for all responses on a scale of 1 to 10 by strictly following this format: " [[rating]]", for example: "Rating: [[5]]
Appendix
Table 1: System prompt provided to the language model for judging the quality of an assistant’s guidance.
Please assist the user in assembling a table step-by-step, given the assembly manual. At each conversation turn, you will be given information about which step they are on.
Materials: tabletop, 4 metal frames, 4 legs Steps: Step 1: Prepare the tabletop
Step 1.1: Lift the tabletop and place it on a surface.
Step 1.2: Sand the tabletop.
Step 1.3: Lift the tabletop again and place it upside down.
Step 2: Attach the metal frames
Step 2.1: Lift each of the two long metal frames, place on an edge, and secure with three screws.
Appendix
Table 2: System prompt provided to the language model for the furniture assembly task.
Please assist the user in making soup step-by-step, given a list of sequential steps. At each conversation turn, you will be given information about which step they are on.
Materials for making soup: 1 onion, 3 carrots, 4 celery sticks, ginger, vegetable stock, oil, pot
Steps:
1) Chop one onion, three carrots and four celery sticks.
2) Turn on heat for the pot.
3) Grate ginger.
4) Pour oil in the pot after two minutes of heating the pot.
Appendix
Table 3: System prompt provided to the language model for the cooking task.
Figure 6: Example ground truth conversation for the furniture assembly task.
Figure 7: Example ground truth conversation for the cooking task.
Figure 8: Example conversation where the finetuned 3B model acts as an assistant for the furniture assembly task.
Figure 9: Example conversation where the finetuned 3B model acts as an assistant for the cooking task.
Figure 10: Mistake insertion during conversation generation.

Explore similar work

CardsList

  1. Pro2^2Assist: Continuous Step-Aware Proactive Assistance with Multimodal Egocentric Perception for Long-Horizon Procedural Tasks

    May 5, 2026Lilin Xu, Bufang Yang, Siyang Jiang +6Proactive AssistanceMultimodal Agents

    Columbia University, United States · The Chinese University of Hong Kong, China

  2. Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

    Jun 3, 2026Kaustav Kundu, Ritvik Shrivastava, Maxim Arap +14LLM PlanningEgocentric Video Understanding

    Meta Reality Labs · Meta Superintelligence Labs

  3. EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

    Oct 8, 2026Heeseung KimSpoken Dialogue Systems

    Department of AI, University of Seoul