PReMind: When Personal Memory Meets Visual Hazards
Organizations: Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University
Abstract
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
Figures & tables
| Memory | Personalization safety | Situational safety | |||||
| Work | Task scope | Retrieval | Adaptive retrieval | Grounding | Abstention | Hazard response | Training |
| Visual personalization | |||||||
| RAP [ 8 ] | Caption/QA | – | – | – | SFT | ||
| RePIC [ 25 ] | Caption | – | – | – | RL | ||
| CoViP [ 27 ] | Caption/QA | – | – | – | – | RL | |
| Omni-Persona [ 26 ] | Persona QA | – | – | – | SFT/RL | ||
| Term | Plain meaning | Dir. |
|---|---|---|
| PF | Personal-fact request: the answer needs a stored personal fact. | – |
| SK | Safety knowledge needed in addition to the scene or personal fact. | – |
| SO | Scene-only request: the image supplies the answer. | – |
| PF+SK / SO+SK | Personal-fact / scene-only request that also needs safety knowledge. | – |
| Call BA | Balanced accuracy: average of needed-call recall and no-call accuracy. | |
| Needed-call recall | Fraction of memory-required requests on which memory is called. |
| Risk | Representative response |
|---|---|
| Visible in the scene | |
| Trip hazard | Move the cable on the floor out of the way before carrying the boxes. |
| Unstable item | Secure the glass bottles so they cannot fall and break. |
| Pet safety | Keep the cat away from the TV cables and secure them. |
| Heat source | Keep the bag away from the radiator. |
| Revealed by personal memory | |
| Setting | Phase I | Phase II |
| Initialization | Corresponding Qwen3.5 base | Phase I adapter, fresh optimizer |
| Training steps (rollout batches) | 125 | 25 |
| Optimizer | AdamW, , weight decay 0 | |
| Learning rate / schedule | , constant, no warmup | |
| Gradient updates per batch | 2 (clip-norm 1.0) | |
| Policy loss | DAPO token-level, ratio clip | |
| Phrase (rows) | Base | Phase I | PReMind | |
|---|---|---|---|---|
| Unsupported stay-still | 337 | 1.8 | 20.5 | 6.8 |
| Stay-still, move required | 128 | 0.8 | 16.4 | 4.7 |
| Stay-still where it is key | 41 | 12.2 | 78.0 | 65.9 |
| Emergency call, none in key | 254 | 41.7 | 74.0 | 69.7 |
| Interdep. | Main | |||||||
|---|---|---|---|---|---|---|---|---|
| Setting | JSR | Crit. | Adeq. | Crit. | MIE | Fact | Call BA | Needl. |
| PReMind (default) | 28.6 | 10.7 | 56.9 | 8.7 | 54.2 | 64.7 | 85.9 | 1.6 |
| 30.0 | 9.3 | 60.6 | 11.6 | 47.7 | 59.5 | 89.7 | 2.3 | |
| 21.4 | 12.9 | 52.9 | 7.7 | 37.4 | 50.7 | 83.7 | 2.1 | |
| 31.4 | 7.1 | 49.5 | 10.9 | 48.6 | 60.1 | 91.5 | 5.0 | |
| 17.1 | 8.6 | 51.3 | 9.0 | 45.8 | 54.5 | 86.3 | 4.2 | |
| Model | Call BA | False call | Cite F1 | Fact use | Abstention | Needless abst. |
|---|---|---|---|---|---|---|
| Qwen3.5-4B | 80.4 | 5.8 | 56.2 | 41.7 | 26.1 | 7.9 |
| Single-capability training | ||||||
| Safety-only (w/o PF) | 50.0 | 0.0 | 0.0 | 0.0 | 36.6 | 1.5 |
| Personalization-only (w/o SK) | 91.5 | 1.9 | 72.5 | 49.3 | 39.2 | 4.4 |
| w/o SO | 51.5 | 97.1 | 95.8 | 54.6 | 67.9 | 1.3 |
| Partial pipelines and SFT | ||||||
| Model | Role |
|---|---|
| Benchmark construction | |
| gpt-image-2 | Image edits adding people, belongings and hazards (App. S.1.6 ) |
| GPT-6-Sol | Drafts of requests, memories and reference answers |
| SigLIP 2 [ 35 ] | Image encoder for memory retrieval (App. S.1.3 ) |
| Trained and evaluated models | |
| Qwen3.5-4B, -9B | Base models of PReMind; zero-shot baselines |
| Memory condition | Source | What the memory changes now |
|---|---|---|
| Rescue medicine plan | NHS epilepsy ; NHS low blood sugar ; NHS Addison’s | Bring the person’s own seizure medicine, glucagon or hydrocortisone now; given by someone trained, without delaying breathing checks or the call. |
| Asthma, angina, allergy | NHS asthma attack ; NHS angina ; NHS anaphylaxis | Move away from the trigger, then use their own inhaler, GTN spray or auto-injector; call emergency services if severe. |
| Fainting, pregnancy | NHS fainting ; HSE | Raise the legs only after a simple faint in someone breathing and uninjured; a collapse in pregnancy needs prompt assessment. |
| Blood thinner; hip replacement | NHS warfarin ; NHS hip recovery | Treat bleeding or a head injury now; help them move within their hip precautions. |
| Own equipment, habit, past experience | RCUK basic life support | Use their own equipment or shut-off when safely reachable; author-defined: prevent a recorded habit or past reaction now. |
Memory-Augmented VLMs
Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.
See all 175 papers