ORACLE: Orchestrate NPC Daily Activities using Contrastive Learning with Transformer-CVAE

Seong-Eun Hong; JuYeong Hwang; RyunHa Lee; HyeongYeop Kang

cs.GRMar 25, 2026

ORACLE: Orchestrate NPC Daily Activities using Contrastive Learning with Transformer-CVAE

Authors: Seong-Eun Hong, JuYeong Hwang, RyunHa Lee, HyeongYeop Kang

Organizations: Korea University, Seoul, South Korea

Abstract

The integration of Non-player characters (NPCs) within digital environments has been increasingly recognized for its potential to augment user immersion and cognitive engagement. The sophisticated orchestration of their daily activities, reflecting the nuances of human daily routines, contributes significantly to the realism of digital environments. Nevertheless, conventional approaches often produce monotonous repetition, falling short of capturing the intricacies of real human activity plans. In response to this, we introduce ORACLE, a novel generative model for the synthesis of realistic indoor daily activity plans, ensuring NPCs' authentic presence in digital habitats. Exploiting the CASAS smart home dataset's 24-hour indoor activity sequences, ORACLE addresses challenges in the dataset, including its imbalanced sequential data, the scarcity of training samples, and the absence of pre-trained models encapsulating human daily activity patterns. ORACLE's training leverages the sequential data processing prowess of Transformers, the generative controllability of Conditional Variational Autoencoders (CVAE), and the discriminative refinement of contrastive learning. Our experimental results validate the superiority of generating NPC activity plans and the efficacy of our design strategies over existing methods.

Figures & tables

Categorized Classes	Criteria
Sleep	5 to 12 hours a day, Once a day
Outing	Up to 12 hours a day, No restrictions
Rest	Up to 12 hours a day, No restrictions
Work	Up to 12 hours a day, No restrictions
Hygiene	Up to 1.5 hours per occurrence, At least once a day
Toilet	Up to 0.5 hours per occurrence, At least once a day

Table 1: The criteria for data filtering.

Major activity classes	Original activity classes
Sleep	sleep, sleep out of bed, go to sleep, nap, wake up
Outing	leave home, enter home, step out, single leave, single enter, staff leave, staff enter, single step out
Rest	watch TV, entertain guests, read, relax, phone, exercise, pet activity, sew
Work	work, work on computer, work at table, work at desk
Hygiene	personal hygiene, groom, bathe, shower
Toilet	toilet, bed toilet transition

Table 2: 42 activity classes in Apartment dataset and 46 activity classes in Home dataset are merged into 12 major activities.

Figure 1: The majority of users responded that the proposed filtering criteria for each activity are suitable for use as the minimum conditions to evaluate the activity’s plausibility.

Figure 2: The workflow of ORACLE framework.

Datasets	Model	WD $\downarrow$	LLM	REAL-	Distinct-10-	Distinct-15-	SAM $\downarrow$	SAM-	REAL-	Distinct-10-	Distinct-15-
			score $\uparrow$	Random $\uparrow$	Random $\uparrow$	Random $\uparrow$		90 $\downarrow$	Condition $\uparrow$	Condition $\uparrow$	Condition $\uparrow$
Apartment dataset	Bowman	0.6263	0.4750	0.4715	0.1899	0.3262	-	-	-	-	-
	Vaswani	-	-	-	-	-	0.5423	1.0491	0.3430	0.1760	0.2809
	Koushik	-	-	-	-	-	0.3390	0.7099	0.6131	0.2633	0.4032
	ORACLE	0.4835	0.5240	0.8416	0.2106	0.3401	0.4218	0.6899	0.9423	0.2831	0.4327
Home dataset	Bowman	1.0470	0.4600	0.1267	0.1699	0.2782	-	-	-	-	-

Table 3: Performance benchmark of ORACLE against three existing models.

Datasets	Model	WD $\downarrow$	LLM	REAL-	Distinct-10-	Distinct-15-	SAM $\downarrow$	SAM-	REAL-	Distinct-10-	Distinct-15-
	Model		score $\uparrow$	Random $\uparrow$	Random $\uparrow$	Random $\uparrow$		90 $\downarrow$	Condition $\uparrow$	Condition $\uparrow$	Condition $\uparrow$
Apartment dataset	ORACLE -C	0.7161	0.4879	0.5942	0.1582	0.2518	0.4233	0.6964	0.9058	0.2662	0.4098
	ORACLE -T	0.5281	0.4849	0.4985	0.1861	0.3211	0.4278	0.6867	0.7153	0.2731	0.4237
	ORACLE	0.4835	0.5240	0.8416	0.2106	0.3401	0.4218	0.6899	0.9423	0.2831	0.4327
Home dataset	ORACLE -C	1.2898	0.4220	0.4833	0.1300	0.2188	0.4744	0.8262	0.8500	0.2984	0.4482
	ORACLE -T	1.0463	0.4430	0.4200	0.1719	0.2719	0.4813	0.7896	0.5633	0.3300	0.4934

Table 4: Ablation study results.

Figure 3: (a) Unity virtual environment where an NPC executes activities. (b) Samples of the visualized plan.

Figure 4: User study evaluation results. (a) Random generation scenario results in Apartment and Home datasets. (b) Conditional generation scenario results in Apartment and Home datasets. Real data is sampled from the test dataset.

Figure 5: (a) Latent visualization of ORACLE and ORACLE -C. (b) Attention map of ORACLE and ORACLE -C.

Panel of Figure 6 — Figure 6: Attention map visualizations of ORACLE and ORACLE -C.

Figure 7: The visualization of daily activities generated by our model.

Input prompts

I’d like to design daily human activities with a defined set {Sleep, Outing, Rest, Work, Hygiene, Toilet, Dress, Cook, Meal, Chore, Snack, Medicine}. I need you to establish numerical parameters for these activities. These parameters must be commonly used for all activities. These parameters should relate to the sequence of activities throughout the day without being tied to specific times. Please recommend parameters and their desired values, drawing on general knowledge about realistic human daily activities.

GPT outputs

Given the scope of your request, I will focus on two parameters that can be universally applied to the activities listed: Duration and Frequency. These parameters are crucial for planning and can accommodate the variability of daily routines. Below are the recommended values based on general knowledge and assumptions about an average adult’s daily activities. Please note that these are guidelines and actual values may vary based on individual needs and lifestyle.

Sleep; Duration: 7-9 hours (Per night), Frequency: 1 time per day Outing; Duration: 1-4 hours (Per outing), Frequency: 1-2 times per day …

Table 5: Input text prompts and GPT outputs for data filtering.

Question 1: Do you consider the criteria presented to be the minimum standards for sleep time and frequency for a normal person’s sustainable life?

Answer 1: People should sleep for at least 5 hours and up to 12 hours per day.

Question 2: Do you consider the criteria presented to be the minimum standards for outing time and frequency for a normal person’s sustainable life?

Answer 2: People should go out for up to 12 hours a day.

Question 3: Do you consider the criteria presented to be the minimum standards for rest time and frequency for a normal person’s sustainable life?

Answer 3: People take a rest for up to 12 hours a day.

Question 4: Do you consider the criteria presented to be the minimum standards for work time and frequency for a normal person’s sustainable life?

Table 6: Survey questions assessing the filtering criteria for each activity.

Input prompts

We want to evaluate if the following [number] indoor daily schedules resemble a human’s daily routine. Please rate it on a scale from 0 to 100 for each. Tell me the score with further explanation. And find the average of those scores.

00:00˜00:05 Rest 00:05˜00:15 Chore 00:15˜00:20 Dress 00:20˜00:35 Hygiene 00:35 08:25 Sleep 08:25˜08:30 Toilet 08:30˜12:10 Sleep 12:10˜12:25 Hygiene 12:25˜12:35 Dress 12:35˜12:40 Rest 12:40 13:10 Outing 13:10˜13:50 Rest 13:50˜14:40 Outing 14:40˜16:00 Rest 16:00˜16:20 Work 16:20˜17:55 Rest 17:55˜18:10 Hygiene 18:10˜18:20 Work 18:20˜18:45 Outing 18:45˜20:40 Rest 20:40˜24:00 Outing

\cdots

GPT outputs

Table 7: Input text prompts and GPT outputs for LLM score.

Conditional Generative Modeling

Conditional generative modeling learns to produce data distributions steered by labels, inputs, text, or physical and structural constraints.

See all 387 papers

PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agents
Mar 12, 2026Minjia Wang, Yunfeng Wang, Xiao Ma +9Synthetic Data Generation LLM Agent Trajectory Synthesis
Apple · Harvard University
TrajGenAgent: A Hierarchical LLM Agent for Human Mobility Trajectory Generation
Jun 10, 2026Siyu Li, Toan Tran, Lingyi Zhao +2Trajectory Generation Human Behavior Simulation
Dept. of Computer Science, Emory University, Atlanta, GA, USA · Novateur Research Solutions, Ashburn, VA, USA
OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Space
Apr 24, 2026Zhuding Liang, Tianyi Yan, Dubing Chen +6Diffusion Transformer Multi-Agent Systems
SKL-IOTSC, CIS, University of Macau, Macau, China · Li Auto Inc, Beijing, China

Mar 12, 2026J/K move · Enter open · S save

PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agents

Minjia Wang, Yunfeng Wang, Xiao Ma, Dexin Lv+8

Apple · Harvard University

Synthetic Data Generation LLM Agent Trajectory Synthesis

Open paper AskPDF arXiv

Digital footprints (records of individuals' interactions with digital systems) are essential for studying behavior, developing personalized applications, and training machine learning models. However, research in this area is often hindered by the scarcity of diverse and accessible data. To address this limitation, we propose a novel method for synthesizing realistic digital footprints using large language model (LLM) agents. Starting from a structured user profile, our approach generates diverse and plausible sequences of user events, ultimately producing corresponding digital artifacts such as emails, messages, calendar entries, reminders, etc. Intrinsic evaluation results demonstrate that the generated dataset is more diverse and realistic than existing baselines. Moreover, models fine-tuned on our synthetic data outperform those trained on other synthetic datasets when evaluated on real-world out-of-distribution tasks.

Abstract

Conditional Generative Modeling

Explore similar work

PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agents

TrajGenAgent: A Hierarchical LLM Agent for Human Mobility Trajectory Generation

OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Space