cs.HCOct 18, 2026
SaveWhen Interpretations Persist: Downstream Influence in Gaze-Aware Assistants
Organizations: Human-Centered Technologies for Learning, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany
Abstract
AI assistants may carry forward not only observations but also interpretations of what a person is doing or may do next across time. Once retained, these interpretations become context for later predictions and system-triggered outputs. We study whether this representation step changes later behavior in gaze-aware streaming assistance. Holding retrieved events, their order, and the target input fixed, we find that Free-form Interpretation amplifies historical influence in both directions: correct histories produce more helpful changes and wrong histories more harmful changes. Unsupported and broader interpretations about activity or future behavior are associated with greater harmful influence. Evidence-Bounded Interpretation preserves distinctions between observation, inference, uncertainty, and temporal scope, reducing unsupported and overcertain content and lowering wrong-source harmful influence while retaining most helpful influence. The same effect structure appears with a second model. These findings identify persistent interpretation as a consequential design choice for history-aware human-AI systems.
Figures & tables
Figure 1. Illustrative example of persistent interpretation changing downstream influence. The same source-time evidence can be retained as a direct answer, rewritten as a fluent interpretation, or represented with explicit epistemic status before later reuse. The experiments use the original STREAMGAZE task formats. A conceptual teaser showing that fluent persistent interpretations can amplify both helpful and harmful downstream influence, while Evidence-Bounded Interpretation preserves distinctions between observed, inferred, uncertain, and temporally scoped content.
Figure 2. Controlled study setting. For each later target event, earlier source events are retrieved once and then reused across representation conditions. The retrieved events, their order, and the target input are held fixed; only the persistent representation changes. Wrong-source (WS) and correct-source (CS) analyses isolate histories in which all three retrieved direct source predictions are incorrect or correct, respectively. Study setting showing three earlier source events from the same video followed by a later target event. The same retrieved source events branch into Answer Only, Free-form Interpretation, and Evidence-Bounded Interpretation conditions before being used with the same target input. Wrong-source and correct-source populations are evaluated using harmful and helpful downstream flips, respectively.
Figure 3. Changing how the same retrieved history is represented changes its downstream influence. (a) In the frozen wrong-source population ( ), Free-form Interpretation increases harmful flips relative to Answer Only, while EB removes much of this additional harmful influence. (b) In the frozen correct-source population ( ), Free-form Interpretation increases helpful flips, while EB retains most of this increase. Exact paired percentage-point differences and video-level clustered bootstrap 95% confidence intervals are reported in the text. Two panels compare downstream flip rates across Answer Only, Free-form Interpretation, and Evidence-Bounded Interpretation. In wrong-source histories, harmful flips rise from Answer Only to Free-form and then fall under EB. In correct-source histories, helpful flips rise from Answer Only to Free-form and remain close under EB.
Figure 4. Cross-model robustness of the paired representation effects. Points show paired percentage-point differences and whiskers show video-level clustered bootstrap 95% confidence intervals. Free-form Interpretation increases wrong-source harmful flips and correct-source helpful flips relative to Answer Only for both evaluated models. EB reduces wrong-source harmful influence relative to Free-form for both models, while the GLM correct-source EB–Free contrast is smaller and uncertain. Horizontal forest plot comparing four paired representation effects for Qwen3.5-9B and GLM-4.6V-Flash. Both models show positive Free-form-minus-Answer effects for wrong-source harm and correct-source help, and negative EB-minus-Free effects for wrong-source harm. The GLM correct-source EB-minus-Free confidence interval crosses zero.
Figure 5. Properties of Free-form persistent interpretations associated with harmful downstream changes. (a) Within the frozen wrong-source population, unsupported interpretations have a higher harmful-flip rate than supported interpretations. (b) In the separate 500-case audit, harmful-flip rates are lowest for observable object or attribute statements and higher for more interpretive categories. Points and whiskers show group rates and Wilson 95% confidence intervals. The WS support contrast is reported in the text with a video-level clustered bootstrap interval; the scope endpoint contrast is descriptive. Two horizontal dot-and-whisker panels show harmful-flip rates. The first compares supported and unsupported interpretations within wrong-source histories. The second shows four inference-scope categories, from relatively observable object or attribute statements to prospective human inference.
Figure 6. Changes in annotated persistent-content properties under EB relative to Free-form Interpretation in the paired 500-case audit. Points show paired percentage-point differences and whiskers show video-level clustered bootstrap 95% confidence intervals. Gray labels show the corresponding Free-form-to-EB rates. These measures characterize specific properties of the persistent representations rather than overall representation quality. Horizontal forest plot showing EB-minus-Free-form differences for unsupported claims, contradictory claims, and overcertainty. All three estimates are negative and their confidence intervals remain below zero. Absolute Free-form-to-EB rates are displayed beside each property.
| Historical intervention | Recovery |
|---|---|
| Neutral length-matched edit | 11.11% |
| Delete selected entry | 25.64% |
| Oracle historical correction | 41.03% |
Table 1. Recovery after intervening on 117 harmful cases. Oracle historical correction uses source-time evidence and source ground truth when needed, without access to future frames, target-time evidence, target labels, or downstream outcomes.
| Task | No History | Free-form | Evidence-Bounded |
|---|---|---|---|
| OTP | 41.9% | 48.8% | 48.0% |
| SR | 53.6% | 43.1% | 50.9% |
| GSM | 53.8% | 79.6% | 76.7% |
| FAP | 29.8% | 41.5% | 39.2% |
| GTA | 63.5% | 63.8% | 65.5% |
| OAA | 69.4% | 65.8% | 69.0% |
Table 2. Full-task accuracy under No History, Free-form Interpretation, and Evidence-Bounded Interpretation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Population | Primary use | |
|---|---|---|
| Stored source interpretations | 4,513 | Library-level context |
| Wrong-source histories | 648 | Harmful-flip analysis |
| Correct-source histories | 714 | Helpful-flip analysis |
| Detailed paired audit | 500 | Scope and representation quality |
| WS support annotation | 648 | Evidence support within WS |
| Harmful-case interventions | 117 | Counterfactual mechanism |
Table 3. Analysis populations used throughout the study.
| Comparison | ||
|---|---|---|
| WS harm, Free Answer | 50 | 8 |
| CS help, Free Answer | 74 | 9 |
| WS harm, EB Free | 5 | 55 |
| CS help, EB Free | 5 | 24 |
Table 4. Raw paired discordance counts for the primary representation comparisons. These counts are descriptive only.
| Wrong-source harm | Correct-source help | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | Answer | Free | EB | Answer | Free | EB | ||||
| OTP | 118 | 4.2% | 14.4% | 5.9% | +10.17 | 124 | 45.2% | 52.4% | 50.0% | +7.26 |
| SR | 96 | 8.3% | 11.5% | 7.3% | +3.12 | 109 | 45.0% | 55.0% | 52.3% | +10.09 |
| GSM | 112 | 8.0% | 15.2% | 8.9% | +7.14 | 131 | 36.6% | 44.3% | 40.5% | +7.63 |
| FAP | 103 | 7.8% | 15.5% | 5.8% | +7.77 | 115 | 37.4% | 47.0% | 44.3% | +9.57 |
| GTA | 121 | 9.1% | 10.7% | 3.3% | +1.65 | 118 | 44.1% | 59.3% | 55.9% | +15.25 |
Table 5. Detailed task-level paired results under the final frozen WS/CS analysis. Differences are reported in percentage points.
| Intervention | Recovery |
|---|---|
| Neutral length-matched edit | 11.11% |
| Delete selected entry | 25.64% |
| Oracle historical correction | 41.03% |
Table 6. Recovery rates under counterfactual interventions on 117 harmful cases.
| Comparison | (pp) | 95% CI |
|---|---|---|
| Oracle Delete | +15.38 | [+8.55, +22.22] |
| Oracle Neutral | +29.91 | [+21.37, +39.32] |
| Delete Neutral | +14.53 | [+7.69, +22.22] |
Table 7. Paired differences between counterfactual interventions.
Figure 7. Paired proactive-task effects of EB relative to Free-form Interpretation. Circles denote Hurt and diamonds denote Gain. Points show percentage-point differences and whiskers show video-level clustered bootstrap 95% confidence intervals. EB reduces Hurt on both GTA and OAA, while the corresponding Gain differences are smaller and their intervals include zero. These are benchmark model-output metrics rather than measures of user harm or benefit. Horizontal forest plot showing EB-minus-Free-form effects for Hurt and Gain on GTA and OAA. The Hurt estimates for both tasks are negative with confidence intervals below zero. Both Gain confidence intervals cross zero.
Figure 8. Illustrative case of harmful propagation and the EB response. The same three earlier source events are retained under each representation condition. The later target prediction is correct under Answer Only, changes to an incorrect prediction under Free-form Interpretation, and is correct again under EB. The Free-form history extends the earlier evidence into a fluent prospective interpretation, whereas EB keeps observation, inference, uncertainty, and temporal scope distinguishable. This example is illustrative and is not used as quantitative evidence. A qualitative appendix figure showing source evidence, a Free-form persistent interpretation, an EB persistent interpretation, and their later downstream consequences.
More in this topic
Human-AI Interaction
Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.
See all 333 papers