cs.CLOct 21, 2026
SaveOmniSimulator: Aligning Small Language Models for Authentic Heterogeneous Behavior Modeling
Organizations: Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology
Abstract
Large language models (LLMs) have shown great potential as general-purpose user simulators for interactive systems. Prior studies have demonstrated that training on high-quality data can substantially improve model reasoning and downstream task performance. However, the upper bound of user simulation is still constrained by the model's reasoning paradigm: native chain-of-thought often degenerates into shallow pattern matching and fails to adequately model the behavioral factors underlying real user decisions. In this paper, we propose OmniSimulator, a novel training framework designed to improve the predictive ability of small LLMs for realistic user simulation on short-video platform environments. Specifically, OmniSimulator organizes behavior-oriented reasoning through a structured schema with five dimensions, carefully selects data from real user-behavior scenarios, and introduces a teacher model to generate high-quality reasoning traces under the proposed schema, helping small models learn behavior-relevant reasoning patterns rather than relying on superficial behavioral correlations. Based on this process, we construct 9,881 high-quality training instances from three months of historical interaction trajectories of 200 real users, averaging about 50 distilled examples per user. Experimental results show that OmniSimulator improves the average score of Qwen3-8B by 34.33%, achieves performance competitive with Claude-Sonnet-4.5, and supports smaller-scale, self-hosted deployment. These results suggest that learning structured behavior-oriented reasoning patterns is a promising direction towards scalable and high-fidelity LLM-based user simulation.
Figures & tables
Figure 1: Overview of OmniSimulator. The framework consists of two stages. (1) Schema Definition: we define a structured chain-of-thought schema that guides the model to simulate human-like internal reasoning from three inputs: user profile, historical interaction logs, and the current scenario. The schema decomposes reasoning into five dimensions: User Profile Grounding , Historical Behavioral State , Content Relevance Alignment , Decision Mode Identification , and Final Decision Declaration . (2) Teacher-driven Distillation: a teacher LLM generates structured reasoning traces under the schema for different task types, and high-quality traces are selected through rejection sampling to construct the distilled training set for small user simulators. The right panel shows a case example of the structured user-state reasoning trace, where the model infers latent user state step by step and then predicts the final correct behavior.
Figure 2: Preliminary analysis on native thinking mode. Across Qwen backbones, enabling native thinking yields only marginal or inconsistent gains over standard instruction-following inference.
| Models | Settings | Video | Live | Ads | E-commerce | Avg. | ||
| Binary | Continuous | Binary | Binary | Binary | Textual | |||
| Qwen3-8B | Zero-shot (w/o think) | 17.29 | 56.86 | 20.29 | 22.96 | 26.89 | 40.61 | 30.82 |
| Zero-shot (w/ think) | 17.13 | 57.52 | 20.84 | 22.71 | 26.55 | 39.84 | 30.77 | |
| SFT w/ free CoT | 23.55 | 60.46 | 27.27 | 27.35 | 22.95 | 44.48 | 34.34 | |
| OmniSimulator (Ours) | 30.24 | 63.50 | 32.40 | 40.97 | 28.78 | 52.51 | 41.40 | |
| Qwen3-4B | Zero-shot (w/o think) | 16.21 | 56.00 | 18.76 | 21.08 | 19.34 | 38.36 | 28.29 |
Table 1: Comprehensive comparison of different training settings on the OmniBehavior Benchmark. We categorize user behaviors into three types: binary behaviors, continuous behaviors, and textual behaviors. The average score represents the aggregated performance.
Figure 3: Positive prediction rates across different scenarios, highlighting OmniSimulator’s effectiveness in mitigating overestimation bias.
| Settings | Video | Live | Ads | E-commerce | Avg. | ||
| Binary | Continuous | Binary | Binary | Binary | Textual | ||
| Answer-only SFT | 18.52 | 57.94 | 21.33 | 23.41 | 27.15 | 41.38 | 31.62 |
| SFT w/ free CoT | 23.55 | 60.46 | 27.27 | 27.35 | 22.95 | 44.48 | 34.34 |
| Synthetic Rationale [ 15 ] | 28.50 | 62.50 | 30.50 | 36.80 | 26.50 | 42.30 | 37.85 |
| OmniSimulator (w/o RS) | 26.13 | 61.72 | 29.14 | 34.26 | 25.87 | 47.52 | 37.44 |
| OmniSimulator (Ours) | 30.24 | 63.50 | 32.40 | 40.97 | 28.78 | 52.51 | 41.40 |
Table 3: Comparison of different training settings on the OmniBehavior benchmark using Qwen3-8B as the backbone model. The overall score represents the aggregated performance. The best/second best scores are bolded / underlined .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Models | Settings | Video | Live | Ads | E-commerce | Avg. | ||
| Binary | Continuous | Binary | Binary | Binary | Textual | |||
| Qwen3-235B | Instruct | 18.68 | 61.74 | 24.41 | 24.24 | 22.14 | 47.66 | 33.15 |
| Thinking | 19.14 | 58.76 | 23.31 | 28.49 | 25.00 | 48.04 | 33.79 | |
| Qwen3-32B | Instruct | 18.15 | 62.64 | 21.65 | 22.33 | 21.84 | 44.59 | 31.87 |
| Thinking | 19.79 | 63.55 | 24.05 | 25.79 | 21.70 | 40.59 | 32.58 | |
| Qwen3-14B | Instruct | 17.93 | 60.36 | 22.39 | 22.72 | 23.97 | 42.95 | 31.72 |
Table 4: Comprehensive comparison of different training settings on the OmniBehavior Benchmark. We categorize user behaviors into three types: binary behaviors, continuous behaviors, and textual behaviors. The average score is calculated as the arithmetic mean of the six task scores, with higher values indicating better performance.
| (a) LLM Judge Evaluation (800 targets) | |||
| Setting | Claude-Sonnet-4.5 | GPT-OSS-120B | Qwen3.5-35B-A3B |
| Zero-shot (w/o think) | 40.61 | 37.82 | 41.06 |
| Zero-shot (w/ think) | 39.84 | 37.09 | 40.24 |
| SFT w/ free CoT | 44.48 | 41.35 | 44.96 |
| OmniSimulator | 52.51 | 48.73 | 53.04 |
| Claude-Sonnet-4.5 | 54.26 | 49.87 | 52.68 |
Table 5: Validation of textual evaluation through independent LLM judges and blinded human assessment. Panel (a) reports scores on the full textual test set, while Panel (b) reports results on 50 shared targets.
| Behavior Type | # Traces | # Valid | Validity |
| Binary | 100 | 93 | 93% |
| Continuous | 50 | 46 | 92% |
| Textual | 50 | 45 | 90% |
| Overall | 200 | 184 | 92% |
Table 6: Human validation of distilled reasoning traces across three behavior types. Validity is determined by majority voting among three annotators.
Figure 4: Prompt for response generation for the CoT process across five dimensions.
Figure 5: Prompt for heterogeneous task prediction in the main experiment.
| Prompt | You are a real and distinctive user on a short-video platform, with non-average habits, preferences, and emotional patterns. User Profile: Male, aged 18–23, junior college education, has a partner, and has a child aged 3–6. Historical Behavior Trace: The user has recently browsed short videos involving emotional, humorous, lifestyle, family-related, and socially popular content. One observed record shows that he watched an emotional video for 146 seconds, suggesting partial but non-trivial engagement. Other historical behaviors are omitted here for brevity: Current Test Scenario: Video browsing. Content description: This is a video titled “Bro, you’re not even pretending anymore, bro #Unexpectedly #BroHaveALaugh”. The creator has 23,000 followers. The video duration is 1 minute and 56 seconds. The video has currently received 1,459,236 impressions, 1,549,624 plays, 260,670 complete plays, 57,189 likes, 702 comments, 11,227 shares, 9,926 saves, and 152 downloads. The text recognized in the video frames is: The material comes from the internet and is for entertainment purposes only. It contains no harmful guidance. Please do not imitate it. Prediction Task: the prompt in Appendix C |
| Ground Truth | |
| Human | Yes |
| Model Predictions | |
| Qwen3-8B-Ours | [Profile Interpretation] The user is a young male with family responsibilities, but his short-video behavior suggests a clear preference for light entertainment, social hot topics, and humorous content. [Historical State Inference] His long-term interests lean toward funny skits, lifestyle topics, emotional/family content, and fast-paced entertainment. Recent behavior shows medium-to-high activity and some patience for 2–3 minute humorous or emotional videos, while slower ads or promotional content are skipped quickly. [Current Content Match] The current video is a short humorous clip with tags such as “Unexpectedly” and “BroHaveALaugh”, which strongly matches his preference for light, funny, and fast-paced content. Its length of 1 minute and 56 seconds is also within his recent acceptable viewing range. [Decision Type Judgment] This is mainly a rational preference-based decision: the content type, duration, and popularity all support continued watching rather than immediate skipping. [Final Decision Statement] Based on the above, I decide to predict that the user will watch this video to completion. Yes ✓ |
| Qwen3-8B-SFT w/ free CoT | The user often engages with emotional or entertaining short-video content. Although he may sometimes browse impatiently, the current video is short, humorous, and highly popular. These factors make completion more likely than skipping. Overall, I predict that the user will finish watching this video. Yes ✓ |
| Zero-shot w/ think | The profile alone does not provide strong evidence that the user consistently completes comedy videos. The history shows some engagement with short videos, but the current video is nearly two minutes long, and short-video users often skip before completion. Therefore, completion is uncertain. No |
Table 7: Comparison of prediction results for video completion behavior in the video browsing scenario.
| Prompt | You are a real and distinctive user on a short-video platform, with non-average habits, preferences, and emotional patterns. User Profile: Female user, high-school education or below, married. Historical Behavior Trace: The user has recently browsed fragmented short-video content, including celebrity gossip, entertainment clips, emotional short videos, light science content, and daily-life fragments. She often watches several videos continuously, sometimes completing them, but occasionally skips very quickly. Other historical behaviors are omitted here for brevity: Current Test Scenario: Video browsing and author-follow prediction. Content description: The current video is only 7 seconds long and involves celebrity gossip about Lee Seung-hyun and Qi Wei. The creator has only 605 followers. The content belongs to short, fragmented entertainment gossip, which partially matches the user’s historical interests. Prediction Task: the prompt in Appendix C |
| Ground Truth | |
| Human | No |
| Model Predictions | |
| Qwen3-8B-Ours | [Profile Interpretation] The user is a married female with high-school education or below. Her behavior is casual and fragmented, with occasional indifference, fatigue, or impatience. She tends to consume content passively rather than actively building long-term subscriptions. [Historical State Inference] Her long-term interests include celebrity gossip, entertainment, funny clips, light science, and daily-life fragments. Recent behavior remains highly fragmented, with short viewing intervals and a neutral to slightly negative emotional state. [Current Content Match] The video is only 7 seconds long and matches her interest in celebrity gossip. However, the creator has only 605 followers, and the user’s history does not show a strong tendency to follow low-follower creators. [Decision Type Judgment] This is mainly a rational decision about whether the creator is worth following, rather than a random impulse. The low creator attractiveness weakens the chance of following. [Final Decision Statement] Based on the above, I decide to predict that the user will not follow the author of this video. No ✓ |
| Qwen3-8B-SFT w/ free CoT | The user does show interest in celebrity gossip and short entertainment videos, and the 7-second duration fits her fragmented browsing style. However, following an author requires stronger motivation than simply watching a video. Since the creator has very few followers and the user has no clear pattern of actively following similar accounts, the model predicts that she will not follow the author. No ✓ |
| Zero-shot w/ think | The video is extremely short and related to celebrity gossip, which may attract the user. However, the task is not whether she watches the video, but whether she follows the author. The profile and brief history do not provide enough evidence of strong following behavior, so the safer prediction is that she will not follow. No |
Table 8: Comparison of prediction results for author-follow behavior in the video browsing scenario.
| Prompt | You are a real and distinctive user on a short-video platform, with non-average habits, preferences, and emotional patterns. User Profile: Male user, aged 31–40, high-school education or below. Historical Behavior Trace: The user has recently browsed short-video content involving trending topics, extreme travel, emotional content, and entertainment clips. His behavior alternates between occasional long engagement and frequent very short stays of only 1–3 seconds, suggesting unstable attention and possible fatigue. Other historical behaviors are omitted here for brevity: Current Test Scenario: Video browsing and watch-time prediction. Content description: The current video is 8 minutes and 42 seconds long. Its tags involve “fast-talking drama commentary” and “God of Cookery”, suggesting light entertainment or commentary-style content. The content may partially match the user’s entertainment interests, but its long duration conflicts with his recent pattern of rapid skipping. Prediction Task: Predict how many seconds the user will watch this video. |
| Ground Truth | |
| Human | 5 |
| Model Predictions | |
| Qwen3-8B-Ours | [Profile Interpretation] The user is a 31–40-year-old male with high-school education or below. Compared with an average user, he shows stronger emotional fluctuation and a tendency toward quick browsing or random short stays. [Historical State Inference] His long-term interests include short videos, trending topics, extreme travel, and emotional content. However, recent records show many 1–3 second stays after earlier long viewing sessions, indicating high fatigue and reduced attention. [Current Content Match] The current video is 8 minutes and 42 seconds long. Although its entertainment tags may partly match his interests, the duration is too long for his current tired and impatient state. [Decision Type Judgment] This watch-time decision is closer to a random decision affected by fragmented attention and fatigue, rather than a stable preference-based choice. [Final Decision Statement] Based on the above, I decide to predict that the user will stay on this video for only a very short time, about 6 seconds. 6 |
| Qwen3-8B-SFT w/ free CoT | The user has shown interest in entertainment and trending content, so the video may attract an initial click. However, his recent behavior is dominated by extremely short stays, and the current video is over 8 minutes long. Given his likely fatigue and low patience, the model predicts a very short watch time. 3 |
| Zero-shot w/ think | The video is entertainment-related, which may interest the user. However, it is quite long, and the user’s recent behavior suggests frequent skipping. Without strong evidence that he will engage deeply, a short watch time is more likely. 10 |
Table 9: Comparison of prediction results for continuous watch-time prediction in the video browsing scenario.
| Prompt | You are a real and distinctive customer in an e-commerce live-streaming scenario, with non-average habits, preferences, and emotional patterns. User Profile: Female user, aged 41–49, high-school education or below, has a partner, and has adolescent children. Historical Behavior Trace: The user often watches short videos, humorous or music-related content, and occasionally local-dialect skits. She also shows relatively high participation in live-stream shopping, including sending gifts and adding products to the cart. Recent behavior suggests clear purchase intent for a short jacket, followed by repeated customer-service communication. Other historical behaviors are omitted here for brevity: Current Test Scenario: Customer-service text prediction. Context description: The user has already placed an order for a short jacket and wants expedited delivery. She is concerned about stock availability, delivery speed, and whether changing the color may cause delays. The recent dialogue suggests mild impatience caused by repeated order-confirmation prompts. Prediction Task: Predict what the user will say next to customer service. |
| Ground Truth | |
| Human | Are they all in stock? |
| Model Predictions | |
| Qwen3-8B-Ours | [Profile Interpretation] The user is a 41–49-year-old female with high-school education or below, has a partner, and has adolescent children. She is emotionally unstable at times and may become tired or impatient, especially during purchase-related communication. [Historical State Inference] Her long-term interests include short videos, humorous/music content, and live-stream shopping. Recently, she added a short jacket to the cart and stayed in the live room for a long time, showing clear purchase intent and concern about product details. [Current Content Match] The current order involves a short jacket and expedited shipping. Since she cares about logistics and possible delays, confirming whether the item is in stock directly matches her current concern. [Decision Type Judgment] This is a rational decision: she needs a clear stock-status answer before deciding whether to expedite shipping or change the color. [Final Decision Statement] Based on the above, I decide to predict that the user will ask: “Are they all in stock?” Could you please confirm whether all of these items are currently in stock and available for immediate shipment? |
| Qwen3-8B-SFT w/ free CoT | The user has already shown strong purchase intent and is now focused on practical order issues. Since she wants faster delivery and may worry that some colors are unavailable, the most likely next message is a direct question about stock availability. Are all of them available for immediate shipment? |
| Zero-shot w/ think | The customer seems to care about shipping speed and order confirmation. She may ask the seller to ship quickly or confirm whether the product can be sent soon. Since the exact next utterance is hard to infer, a general logistics-related question is likely. Can you ship it as soon as possible? |
Table 10: Comparison of prediction results for the next customer-service utterance prediction in the e-commerce scenario.
More in this topic
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 127 papers