COOL: Curiosity-Driven Object Ownership Learning for Personalized Robotic Assistance

Samira Huber; Ruben Hammele; Sören Pirk

cs.ROOct 7, 2026

COOL: Curiosity-Driven Object Ownership Learning for Personalized Robotic Assistance

Authors: Samira Huber, Ruben Hammele, Sören Pirk

Organizations: Department of Computer Science, Kiel University, Germany

Abstract

Robots are increasingly expected to provide personalized services in everyday environments. To do so, they must ground natural-language commands such as "Where is my backpack?" or "Find my bottle" and execute them by reasoning about object instances, people, locations, and ownership. This is challenging because ownership is rarely labeled explicitly and must be inferred from long-term, behavioral evidence of human-object interactions. To address this, we present COOL, a novel robotic framework for autonomously learning object ownership from everyday observations and maintaining a long-term spatial memory of its environment. To keep its memory current, COOL uses an agent-based curiosity-driven data collection strategy that guides the robot toward the most promising locations to gain information and refresh stale observations. Offline experiments, ablation studies, and real-world evaluations show that COOL can infer ownership relations from real-world interactions and use this knowledge for ownership-conditioned navigation and task execution.

Figures & tables

Figure 1: COOL enables personalized object search by combining long-term memory, ownership inference, and spatial grounding. Given a natural-language request such as “Find my backpack” (left), the robot first retrieves prior observations linking people and object instances, then infers the likely owner from accumulated human–object interaction evidence (middle), and finally grounds the selected object in the map to navigate to its most likely location (right).

Figure 2: Our framework integrates curiosity-driven data collection with persistent spatial memory and ownership-conditioned reasoning. A curiosity-driven agent (top) decides when to stay and observe or move to more informative locations, the memory module (blue) clusters repeated person and object detections and stores human–object interactions, while a user agent (red) retrieves this evidence to infer ownership and execute personalized commands such as Find my bottle .

Figure 3: Agent-based reasoning in COOL: we use two language agents to operate on a shared long-term spatial memory. A user agent (red) retrieves people, objects, locations, and interaction histories to infer ownership and generate navigation goals, while a curiosity-driven exploration agent (blue) decides where to go next to collect ownership-relevant evidence and refresh stale observations.

Variant	Accuracy (%)	Interaction (%)	Location (%)	Lookup (%)	Ownership (%)	Navigation (%)	Func. calls	Time (sec)
Plain LLM	$37.0\pm 0.0$	$61.0\pm 0.0$	$26.0\pm 0.0$	$\mathbf{33.0\pm 0.0}$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.00\pm 0.00$	$4.69\pm 0.07$
Heuristic	$21.0\pm 0.0$	$0.0\pm 0.0$	$78.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$11.0\pm 0.0$	$0.25\pm 0.00$	$0.01\pm 0.00$
No Interaction	$36.0\pm 2.0$	$6.0\pm 3.0$	$\mathbf{96.0\pm 0.0}$	$\mathbf{33.0\pm 0.0}$	$33.0\pm 12.0$	$15.0\pm 6.0$	$4.60\pm 0.15$	$6.85\pm 0.68$
Rule-based	$15.0\pm 0.0$	$18.0\pm 0.0$	$37.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$11.0\pm 0.0$	$0.25\pm 0.00$	$\mathbf{0.01\pm 0.00}$
COOL	$\mathbf{89.0\pm 2.0}$	$\mathbf{94.0\pm 0.0}$	$\mathbf{96.0\pm 0.0}$	$\mathbf{33.0\pm 0.0}$	$\mathbf{100.0\pm 0.0}$	$\mathbf{89.0\pm 0.13}$	$2.18\pm 0.05$	$2.35\pm 0.10$

Table 1: Results on the offline reasoning benchmark. We report overall and category-specific accuracy over three seeds, together with the average number of tool calls and wall-clock time per query.

Robustness Test	Accuracy (%)
Baseline	84.33 $\pm$ 2.75
Detection Dropout (40%)	81.48 $\pm$ 5.30
Detection Dropout (60%)	61.12 $\pm$ 5.78
Detection Dropout (80%)	47.78 $\pm$ 5.77
False-Positive Injection (10%)	78.51 $\pm$ 5.78
False-Positive Injection (50%)	66.67 $\pm$ 5.99

Table 2: Robustness test on sitcom footage. Accuracy (%) as mean

\pm

standard deviation (10 seeds).

Task Type	Ownership	Person Distinction	Person ReID	Object ReID	Long-Term Tracking	Visual Ambiguity	Reachability Reasoning	Accuracy (%)
Room Navigation	$-$	$-$	$-$	$-$	$-$	$-$	$+$	100
Object Relocation	$+$	$-$	$+$	$+$	$+$	$-$	$-$	90
Owner-based Distinction	$+$	$+$	$+$	$+$	$+$	$-$	$-$	100
Visual Ambiguity	$+$	$+$	$-$	$-$	$-$	$+$	$-$	90
Ownership-Preserving Update	$+$	$+$	$-$	$+$	$+$	$+$	$-$	90

Table 3: Real-world navigation tasks and evaluated capabilities. Accuracy is measured over 10 runs per task. A “

+

” denotes a capability that is required for solving the task, whereas “

-

” denotes a capability that is not involved in the task. Details are provided in Appendix D .

Panel of Figure 4 — Figure 4: Three object ownership inference examples. 1) Object Relocation: Find Simba’s ball (top), 2) Owner-based Distinction: Find Nemo’s backpack (middle), 3) Visual Ambiguity: Find Simba’s backpack (bottom).

Strategy	F1	Recall	Prec.	Yield	Moves	Observed	Best
Random	$65.0\%$	$88.2\%$	$52.0\%$	$0.36$	$182$	$65$	$0/15$
Round-Robin	$67.9\%$	$82.9\%$	$57.8\%$	$0.65$	$91$	$59$	$0/15$
Frequency	$69.9\%$	$86.2\%$	$59.5\%$	$0.50$	$146$	$\mathbf{73}$	$0/15$
Staleness	$63.1\%$	$\mathbf{90.6\%}$	$48.7\%$	$0.36$	$182$	$66$	$0/15$
Active Mapping	$63.2\%$	$90.5\%$	$48.8\%$	$0.36$	$182$	$65$	$0/15$
Greedy Hazard	$76.5\%$	$83.3\%$	$71.3\%$	$0.64$	$108$	$69$	$0/15$

Table 4: Navigation strategy comparison across all logs. Moves denotes room-to-room relocations, Observed denotes change events that occurred while the agent was in the room, and Yield is observed changes per move. Recall and Precision are measured over the agent’s per-visit change judgements. Best counts logs on which a strategy attains the highest F1 among those listed. All values are means over 15 logs × 10 seeds.

Figure 5: Real-world curiosity-driven exploration on the Spot robot. Trajectories over a 4.5-hour deployment in a four-room office environment are shown.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value	Description
Embedding similarity
Visual embedding threshold	$\tau_{\text{emb}}$	0.643	Cosine similarity; runtime override (code default 0.85)
Face embedding threshold	$\tau_{\text{face}}$	0.60	Cosine similarity; runtime override (code default 0.68)
Similarity metric	–	Cosine	pgvector <=> cosine distance; $\text{sim}=1-\text{dist}$
Centroid update	–	Arithmetic mean	SQL AVG(oo.embedding) over cluster observations
Attribute part embeddings	–	false	Part-level attribute embeddings are disabled

Appendix

Table 5: Clustering parameters used at inference time for person/object re-identification.

Tool	Input	Output	Purpose
Discovery and search
search_objects_by_class_id	class_id, name, min_obs, limit	object list	Find tracked objects or people by class or name
list_objects	class_name (opt.), limit	object list	Enumerate all tracked objects, optionally filtered
find_person_by_name	person_name	person entity	Resolve a named person to their latest location
Object inspection
get_object_observations	object_id, time filters, limit	observation history	Retrieve the location history of one entity

Appendix

Table 6: Structured tools exposed to the user agent for object search, inspection, interaction retrieval, scene understanding, and navigation.

Tool	Input	Output	Purpose
get_room_visit_history	limit	per-room statistics	Retrieve visit counts, last-visit times, and event counts per room
get_room_event_density	hours, limit	event counts	Count recent interaction events per room over a time window
get_stale_rooms	threshold_min, limit	stale room list	Identify rooms that have not been visited recently

Appendix

Table 7: Structured tools used by the curiosity-driven navigation strategy to select the next room to visit.

Evaluation Category	Questions
Interaction	33
Location	22
Navigation	13
Lookup	6
Ownership	5
Total	79

Appendix

Table 8: Test set composition.

Failure Mode	Parameter	Perturbation
Detection Dropout	$p\in\{0.4,0.6,0.8\}$	Randomly remove observation rows
False-Positive Injection	$p\in\{0.1,0.5\}$	Add observations with incorrect identities
Embedding Noise	$\sigma=0.10$	Add Gaussian noise to embeddings
Track Fragmentation	$p=0.2$	Split tracks and remove identity names
Face Occlusion	$p=0.5$	Remove face image information
Temporal Jitter	$\pm 300$ s	Perturb observation timestamps

Appendix

Table 9: Perturbations used in the robustness evaluation.

Aspect	6-room floor plan	24-room floor plan
Rooms	6: kitchen, storage, office 1–3, meeting room	24: 16 offices, 3 meeting rooms, kitchen, lounge, storage, printer room, server room
Room types	social, utility, office, meeting	social, utility, office, meeting
People	8 employees, each with a home office and a role	40 employees, each with a home office and a role
Objects	12 personal objects with a fixed owner (laptop, red cup, backpack, water bottle, notebook, pen, headphones, coffee mug, tablet, jacket, yoga mat, camera), plus activity-spawned temporary items	40 personal objects with a fixed owner, plus activity-spawned temporary items
Fixtures	per room, e.g. coffee machine, fridge, microwave (kitchen); projector, whiteboard, conference table (meeting room)	as left, plus printer, paper shelf, server rack
Geometry	pairwise room distances $d\in\{1,2,3,4\}$ (adjacent offices $d=1$ , kitchen–office 3 $d=4$ )	same distance model over the larger plan

Appendix

Table 10: Dataset specification. The two floor plans differ in size, population and object count; every other component of the simulator is shared.

Room Type	Activity	Duration
Office	Works at their desk	$20$ – $200$ min
	Types on their laptop	$15$ – $120$ min
	Reads a document	$10$ – $40$ min
	Joins a video call	$15$ – $120$ min
	Writes notes	$10$ – $30$ min
	Checks their phone	$5$ min

Appendix

Table 11: Activities and durations by room type.

World	R	P	O	Obs.	Maj.	I/D	Ep.	$p_{\mathrm{head}}$	Prior / property
6-room worlds
diffused	6	8	21	1098	365	–	–	–	hot; diffused activity
regime	6	8	24	1098	140	–	–	–	hot; daily regime shift, sparse changes
adversarial	6	8	18	1098	296	–	–	–	hot; moving hotspots
base	6	8	9	1098	234	121/267	–	–	hot; v1 parity under v2 generator
distractor	6	8	10	1098	280	135/273	–	–	hot; $4\times$ churn in kitchen/meeting, mostly distractor-class

Appendix

Table 12: Evaluation worlds. Each world spans 3 simulated days, 09:00–14:00 at 5 min intervals (

183

decision steps/run). Obs. = logged room observations; Maj. = ownership-relevant scene changes; I/D = v2 interesting/distractor changes; Ep. = scripted intent episodes;

p_{\mathrm{head}}

= probability that a departure cue names its destination. Occupancy priors are hot , mild , or flat . Each world is fixed; seeds vary policy randomness only.

World	Kitchen	Storage	Office 1	Office 2	Office 3	Meeting	Gini
default	$96$	$16$	$58$	$70$	$43$	$82$	$0.24$
regime	$32$	$13$	$28$	$24$	$25$	$18$	$0.15$
adversarial	$70$	$3$	$54$	$66$	$48$	$55$	$0.22$
base_v2	$62$	$9$	$50$	$36$	$28$	$49$	$0.25$
distractor	$88$	$2$	$38$	$46$	$43$	$63$	$0.30$
intent_cued	$65$	$10$	$43$	$39$	$31$	$62$	$0.25$

Appendix

Table 13: Major changes per room, 3-day totals, 6-room worlds. Gini is the concentration of changes across the six rooms.

World	Changes	Gini	Three busiest rooms (share of major changes)
scaled_base_24room	$1041$	$0.21$	kitchen $11\%$ , meeting room 1 $7\%$ , office 3 $5\%$
scaled_24room	$1046$	$0.27$	kitchen $11\%$ , meeting room 1 $7\%$ , meeting room 3 $6\%$
scaled_distractor_24room	$1087$	$0.28$	kitchen $14\%$ , meeting room 1 $9\%$ , meeting room 3 $7\%$
scaled_heading_24room	$1055$	$0.24$	kitchen $10\%$ , meeting room 1 $7\%$ , meeting room 2 $6\%$
agent_cued_24room	$1081$	$0.19$	meeting room 1 $8\%$ , meeting room 2 $7\%$ , kitchen $7\%$
agent_cued_flat_24room	$1107$	$0.12$	meeting room 1 $7\%$ , lounge $6\%$ , meeting room 3 $5\%$

Appendix

Table 14: Concentration of major changes in the 24-room worlds, 3-day totals. With 24 rooms no single room dominates: the busiest room carries

7

–

14\%

of all major changes, which is what makes camping strategies expensive at scale.

Strategy	Recall	F1	Yield	TP	FP	FN	Moves	Changes	Cov.	Missed
Random	$89.0\%$	$67.3\%$ $\pm 3.6$	$0.43$	$64$	$52$	$10$	$182$	$78$	$45.9\%$	$19$
Round-Robin	$93.2\%$	$81.4\%$ $\pm 0.8$	$0.92$	$74$	$28$	$6$	$91$	$84$	$49.7\%$	$25$
Frequency	$89.3\%$	$72.8\%$ $\pm 5.4$	$0.61$	$71$	$42$	$11$	$139$	$85$	$50.4\%$	$42$
Staleness	$90.4\%$	$67.1\%$ $\pm 2.4$	$0.44$	$68$	$58$	$9$	$182$	$81$	$47.7\%$	$18$
Active Mapping	$88.0\%$	$66.8\%$ $\pm 2.4$	$0.44$	$68$	$57$	$10$	$182$	$81$	$47.8\%$	$\mathbf{15}$
Greedy Hazard	$86.6\%$	$79.4\%$ $\pm 1.0$	$0.65$	$78$	$28$	$12$	$144$	$93$	$55.3\%$	$76$

Appendix

Table 15: Navigation-strategy comparison on default . Mean over 10 seeds;

\pm