ReferClaw: Disambiguating User Intent for Egocentric Assistance
Organizations: National University of Singapore · University of Science and Technology of China
Abstract
Users of egocentric AI assistants often ask short, ambiguous questions whose intended referents cannot be uniquely resolved from video and dialogue history. For example, "Where did I leave it?" may refer to several previously handled objects. Existing VideoQA systems typically commit to a single interpretation, risking plausible but intention-misaligned answers. We introduce ReferClaw, an agentic framework that formulates egocentric streaming QA as an interactive process: the assistant asks clarifying questions to resolve user intent before answering. Three cooperating agents identify candidate interpretations, request clarification, and generate answers grounded in the resolved intent and visual evidence. A bounded, cost-aware policy balances expected information gain against user effort while limiting clarification turns and question complexity. Referclaw learns from interaction through intent-specific skill updates and process-reward-guided policy optimization with agent-level credit assignment. Evaluations on augmented egocentric QA benchmarks demonstrate improved answer accuracy and joint answer-and-grounding performance over existing baselines, alongside zero-shot transfer across datasets. These findings support interactive clarification as a practical approach to resolving online user intent in egocentric streaming assistance.
Figures & tables
| Method | T-mIoU | S-mIoU | Ans. | Acc@GQA |
|---|---|---|---|---|
| Open-o3-Video | 44.7 | 41.2 | 54.0 | 27.0 |
| EG-Reasoner | 46.5 | 39.8 | 54.8 | 27.6 |
| ReferClaw (Skills) | 48.9 | 41.6 | 58.8 | 30.1 |
| ReferClaw (Full) | 50.1 | 42.6 | 60.7 | 31.6 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Action | Output / next state | |
|---|---|---|---|
| T1 | if : Reground , , re-evaluate; else terminate | Abstain | |
| T2 | or | AnswerPath ; if because all plausible candidates share an answer, EFA receives their pooled evidence and is their summed mass | Answer or Abstain |
| T3 | , , CCA returns with and partition | ask ; receive ; ; then (a) uncertain, invalid, or missing: , ; (b) = other/none or unmatched constraint: Reground if , else Abstain ; (c) otherwise by Eq. 4 , drop candidates with ; | continue at T1 |
| T4 | otherwise ( , or no admissible question with positive utility) | if and : AnswerPath and AnswerPath , each labeled by its interpretation; else AnswerPath | Conditional (if both branches pass; Abstain if either fails) or Answer / Abstain |
| Metric | Averaged over | Success / definition |
|---|---|---|
| Ans. Acc. (Overall; type; Chain, Deictic) | all 6,240 (2,036) inputs; that type or subset | judge verdict w.r.t. target intent; Conditional / Abstain wrong |
| Ambiguous / well-specified acc. | 1,197 / 5,043 (418 / –) | Overall |
| T-mIoU, [email protected], Ans. (Tab. 7 ) | 4,655 (1,447) grounding-eligible | mean T-IoU; T-IoU ; judge verdict |
| S-mIoU, [email protected], Acc@GQA | 3,914 (1,203) box-annotated | S-IoU on annotated frame; S-IoU ; correct and T-, S-IoU |
| Amb-F1; Ask-P / Ask-R | all inputs (1,197 ambiguous positives) | F1 of the ambiguity flag; precision / recall of asking |
| Unnecessary-ask rate | 5,043 well-specified (Fig. 3 ); 201 G4 (Sec. 4.4 ) | at least one question asked |
| Grounding | 1 fps causal stream; retrieved clips at 4 fps, ; 8 temporal proposals; detection/text thresholds ( for the single re-grounding pass) |
| Candidates | after merging; ; ; drop threshold ; question pool (3 multiple-choice, 3 open-ended) |
| Thresholds | , , , , |
| Budget | , displayed tokens, options incl. other/none (one-shot stress test: , ); generation cap 32 tokens per candidate question (longer candidates discarded) and 128 answer tokens |
| Respondent | 8 samples per , smoothing , free text mapped to classes by BGE-M3; , ; human, simulator; components normalized to by validation 5th/95th percentiles |
| Learning | PRM weights ; one failure triggers skill synthesis ( failures per call), BGE-M3 top-3 retrieval at cosine , injected skills; LoRA rank 16, , dropout ; query buffer 128; ; KL to Base; clip ; lr , batch 64, one epoch per window; promotion probe 500+500 validation examples (target type , probe , unnecessary-ask and cost ) |
| Decoding | IGA/CCA temperature , top- ; EFA/verifier temperature |
Egocentric VQA
Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.
See all 49 papers