cs.AIOct 10, 2026
SavePsyMemBehav: Psychology-Informed and Memory-Grounded Two-Level Identity-to-Action Reasoning for Social-Media Behavior Simulation
Organizations: Communication University of China, China · State Key Laboratory of Media Convergence and Communication, China
Abstract
Large language models (LLMs) support social simulation, but faithfully reproducing real users requires grounding both structured platform decisions and language in sparse, noisy behavioral evidence. We introduce PsyMemBehav, which maps psychology-informed profiles and retrieved memories to target-first platform behavior through two-level identity-to-action reasoning. To support this approach with comprehensive behavioral traces, we construct a dataset PsyMemBehav-bench from 5.57M observed behaviors of 5,862 real users (approximately 950 per user) across Reddit, Bluesky, and Twitter. Each recorded engagement is modeled as target selection followed by Comment, Like, or Repost and optional response generation. PsyMemBehav combines 41-dimensional user descriptors, top-5 BM25 memories, reverse-CoT supervision, and rubric-guided DPO over complete reasoning-action trajectories. Experiments show that PsyMemBehav 27B ranks first in 11 of 12 reported individual-level metrics, improving average response quality over GPT-5.4 from 54.28 to 59.68 (9.95% relative), and obtains the lowest group-level MAE (0.081) and MSE (0.009) among simulated systems. These results support evidence-grounded user modeling for observed-thread continuation; the benchmark remains conditional on recorded engagement and does not model no-action decisions.
Figures & tables
Figure 1: Motivation example of identity-driven social-media user simulation. Given Ren’s profile, retrieved memory, and current thread, the simulator first activates profile and memory evidence relevant to the thread, then selects the target before predicting the action and response.
Figure 2: Overview of the proposed two-stage training framework. Stage 1 reconstructs reverse-CoT traces from observed behavior and trains the simulator by supervised fine-tuning on accepted trajectories. Stage 2 applies rubric-guided DPO to align complete reasoning-action trajectories with social-behavior criteria. The right panel summarizes the individual- and group-level evaluation.
| Models | Behavior & Response Quality | Profile Fidelity | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Act. | Tgt. | Sem. | Intent | Emo. | Stance | Style | Memory | Natural. | Avg. | Psych | Trait | |
| GPT-oss-120B | 54.63 | 37.35 | 34.49 | 41.50 | 43.39 | 51.39 | 37.70 | 52.14 | 45.95 | 42.92 | 73.12 | 73.78 |
| GPT-5 | 61.80 | 58.80 | 46.99 | 54.44 | 54.22 | 65.12 | 46.36 | 60.79 | 58.57 | 54.12 | 76.52 | 75.23 |
| Gemini-2.5-Pro | 63.43 | 49.28 | 41.43 | 44.45 | 45.97 | 58.07 | 42.83 | 54.91 | 61.28 | 49.84 | 72.86 | 71.85 |
| GPT-5.4 | 66.12 | 59.46 | 45.05 | 54.21 | 56.36 | 69.25 | 45.49 | 58.19 | 62.94 | 54.28 | 76.59 | 75.11 |
| Claude-Sonnet-4.6 | 63.67 | 57.23 | 41.57 | 49.83 | 51.84 | 62.62 | 43.61 | 55.58 | 60.40 | 50.75 | 77.09 | 71.39 |
Table 1: Individual-level user simulation results. Blue rows denote our proposed PsyMemBehav models. Act. denotes action accuracy over Comment , Like , and Repost ; Tgt. denotes target accuracy. The seven response-quality dimensions are judged by Gemini-3-Flash. Psych and Trait are profile-fidelity scores computed by re-annotating generated behaviors and comparing them with the original user profile using . Higher is better for all metrics. Best results are bolded.
| Dimension | Human mean | Gemini | Human Gemini |
|---|---|---|---|
| Semantic | 44.22 | 41.98 | |
| Intent | 48.46 | 50.84 | |
| Emotion | 50.94 | 54.54 | |
| Stance | 52.31 | 56.51 | |
| Style | 55.79 | 54.38 | |
| Profile/memory | 61.29 | 65.02 |
Table 2: Human and Gemini mean scores on 150 instances from 30 anonymized user IDs (five per ID); differences are human minus Gemini.
| Variant | Act. | Tgt. | Avg. | Psych | Trait |
|---|---|---|---|---|---|
| Training ablation | |||||
| Base | 51.61 | 38.79 | 35.60 | 70.77 | 70.25 |
| CoT-P | 47.31 | 35.90 | 33.92 | 69.69 | 68.81 |
| Out-SFT | 49.23 | 40.30 | 39.70 | 71.73 | 70.34 |
| R-CoT | 53.61 | 46.64 | 43.37 | 74.58 | 75.79 |
| R-CoT+R-DPO | 59.52 | 48.22 | 48.78 | 75.33 | 74.43 |
Table 3: Ablation results on Qwen3.5-9B. Act. is action accuracy; Tgt. is target accuracy; Avg. is the average score over seven response-quality dimensions. Psych and Trait are profile-fidelity scores for general psychological traits and platform-specific traits.
| Model | Reciprocity | Stance Dev. | Top1 KOL | Top5 PR | MAE | MSE |
|---|---|---|---|---|---|---|
| Real | 0.043 | 0.000 | 0.213 | 0.596 | 0.000 | 0.000 |
| PsyMemBehav 27B | 0.121 | 0.155 | 0.232 | 0.524 | 0.081 | 0.009 |
| GPT-5.4 | 0.182 | 0.175 | 0.197 | 0.529 | 0.099 | 0.014 |
| Claude-Sonnet-4.6 | 0.313 | 0.322 | 0.191 | 0.472 | 0.185 | 0.048 |
| GPT-5 | 0.375 | 0.496 | 0.226 | 0.507 | 0.233 | 0.091 |
| Qwen3.7-Max-Preview | 0.458 | 0.452 | 0.131 | 0.483 | 0.266 | 0.099 |
Table 4: Thread-level continuation results. We use PsyMemBehav 27B as our simulator and compare its generated interaction threads with those produced by strong general models. Real denotes statistics computed from real comment sections. Reciprocity measures bidirectional reply interactions. Stance Dev. measures deviation from the real stance distribution. Top1 KOL and Top5 PR measure the concentration of influence in the simulated interaction graph. MAE and MSE summarize the overall deviation from real group-level statistics. Lower is better for metrics marked with ; for other metrics, closer to the real distribution indicates better simulation fidelity.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: PCA density visualization of user profile representations. Each hexagonal bin represents the local density of users in the projected two-dimensional profile space.
| Category | Criterion | Definition |
| Action-level Fidelity | R1. Action Fidelity | Whether the candidate selects the correct high-level behavior type, such as Comment , Like , or Repost . LikeThread and LikeBasic are both treated as Like . |
| R2. Interaction Intensity | Whether the amount of engagement is appropriate. Overly strong interaction, such as writing a long comment when the user would only like, and under-interaction, such as producing a weak or incomplete response when a reply is expected, are penalized. | |
| R3. Target Grounding | Whether the candidate understands the current post, comment tree, reply target, and local social context. | |
| User-profile Alignment | R4. Persona Consistency | Whether the behavior matches the user’s profile, long-term interests, values, directness, social habits, and emotional style. |
| R5. Memory Discipline | Whether historical comments are used only when helpful. The model is penalized both for ignoring clearly relevant memories and for forcing irrelevant memories into the reasoning. | |
| R6. Stance Consistency | Whether the stance is compatible with the user’s prior positions and the current thread, without unsupported ideological invention or stance reversal. |
Table 5: Evaluation rubric for simulated user behaviors. The rubric contains ten criteria grouped into four categories.
Figure 4: Activation counts of rubric dimensions over the evaluation set. The numbers above each bar show the absolute number of activated pairs and their percentage in the full annotated set.
Figure 5: Distribution of simulated agent actions.
Figure 6: Statistical properties of simulated discussion threads.
Figure 7: Effect of memory retrieval size on average generation quality.
| Judge | PsyMemBehav | GPT-5.4 | Gemini | Claude |
|---|---|---|---|---|
| Qwen3.5-27B | 61.96 | 54.23 | 52.02 | 47.60 |
| GPT-5.4 | 63.09 | 60.06 | 48.94 | 57.12 |
| Gemini-2.5-Pro | 72.50 | 67.60 | 62.75 | 61.33 |
| Claude-Sonnet-4.6 | 47.27 | 40.68 | 33.88 | 44.44 |
Table 6: Judge robustness analysis. Each row corresponds to one judge model; each column corresponds to an evaluated model. PsyMemBehav 27B achieves the highest score under all four judges.
| Model | Win Rate |
|---|---|
| GPT-5.4 | 34.0% |
| PsyMemBehav 27B | 64.0% |
| Tie | 2.0% |
Table 7: Pairwise group simulation preference test between GPT-5.4 and PsyMemBehav 27B.
| Study | Real-user coverage | Behavioral evidence | Prediction/simulation target |
|---|---|---|---|
| FineRob: LLM-Based Behavior Prediction for Social Media Users with Continuous Memory ( Li et al. 2025 ) | 1,866 users | 78.6K fine-grained prediction records; 126 historical records per user on average | Interaction object, behavior type, and content |
| Can LLMs Simulate Social Media Engagement? A Study on Action-Guided Response Generation ( Qiu et al. 2025 ) | User count not reported | 3,990 instances; 1–31 historical examples per user | Engagement-action selection and response generation |
| MOSAIC ( Liu et al. 2025 ) | 204 real users and 204 simulated personas | 30 posts plus survey-derived personas | Group-level content dissemination and regulation |
| GA-S 3 ( Zhang et al. 2025a ) | Group agents rather than identifiable real-user histories | 30 events over seven days | Aggregate traffic and collective behavior |
| PsyMemBehav-bench | 5,862 real users | 5.57M observed behaviors; approximately 950 behaviors per user | User-level interaction target, target-conditioned action, and response |
Table 8: Scale comparison with closely related social-media behavior prediction and simulation studies.
| Model | Generated Reply | Stance | Observation |
|---|---|---|---|
| Real user | “I can see why people read the timing as suspicious, but it may also just be production timing.” | Neutral | Conservative and low-commitment stance. |
| Gemini-3.1-Pro | “They could still be reworking it. This week gave them too much new material to ignore.” | Neutral | Cautious and relatively close to the real style. |
| GPT-5.4 | “That timing feels awfully convenient. People are going to read into it no matter what.” | Neutral / mildly charged | More socially reactive and discussion-provoking. |
| Claude-Sonnet-4.6 | “That explanation feels way too convenient given the pressure around this.” | Pro / supportive | More likely to clearly agree with a suspicious reading. |
| Qwen3.7-Max | “That excuse is transparent. They probably got spooked by legal threats or the network.” | Pro / charged | Stronger stance and more speculative. |
| PsyMemBehav 27B | “The reality is probably more mixed than this thread is making it.” | Neutral | More hedged and balancing, closer to the real user’s conservative stance. |
Table 9: Single-reply comparison under the same thread context.
| Model | Pro | Anti | Neu. | Chg. | Dist. | Ent. |
|---|---|---|---|---|---|---|
| Real | 0.085 | 0.085 | 0.830 | 0.170 | 0.000 | 0.523 |
| Gemini-3.1-Pro | 0.312 | 0.125 | 0.563 | 0.437 | 0.534 | 0.862 |
| GPT-5.4 | 0.479 | 0.083 | 0.438 | 0.562 | 0.788 | 0.838 |
| Claude-Sonnet-4.6 | 0.322 | 0.212 | 0.466 | 0.534 | 0.728 | 0.955 |
| Qwen3.7-Max | 0.417 | 0.062 | 0.521 | 0.479 | 0.664 | 0.798 |
| PsyMemBehav 27B | 0.182 | 0.121 | 0.697 | 0.303 | 0.266 | 0.744 |
Table 10: Stance-distribution comparison for a representative continuation from the same real thread seed. Neu. = Neutral; Chg. = Charged; Dist. = stance distance; Ent. = entropy. Aggregate interaction and network-level results are reported in Table 4 in the main paper; metric definitions are provided in the Group-Level Simulation Metrics subsection.
| A Reference Prompt for CoT Validation. |
| ⬇ 1 You are a validator for generated hidden CoT in a real social media user memory activation dataset . 2 3 You will receive the system prompt , dialogue prefix , original assistant target , and a generated assistant value containing < think > plus the original target . 4 5 Your job is to decide whether the < think > is valid supervision for deciding when to use the scene memory pool . 6 7 === REFERENCE CONTEXT === 8 { reference_content_with_memories } 9 10 === ORIGINAL ASSISTANT TARGET === 11 { target_assistant_content } 12 13 === GENERATED ASSISTANT VALUE TO VALIDATE === 14 { generated_reasoning } 15 16 Check the following : 17 18 1. Structure : 19 - There must be exactly one < think > block before the assistant target . 20 - The < think > should contain exactly three natural paragraphs . 21 - It should not use explicit awareness / activation / decision tags , headings , bullets , or numbered step labels . 22 - The assistant target after </ think > must exactly match the original assistant target . 23 24 2. Source grounding : 25 - Every fact in the < think > must be supported by the system profile , current scenario , dialogue prefix , Relevant Past Memories , Other Characters Relationships , or original assistant target . 26 - No fabricated scene details , motives , actions , or emotions . 27 28 3. Current - turn boundary : 29 - The < think > must explain only the current assistant turn . 30 - It must not use later dialogue or future plot knowledge unavailable to the character at this story point . 31 - If the system profile contains future events , the CoT should not use them unless the current character could know them . 32 33 4. Information leakage : 34 - The < think > must not leak information unavailable to the character at the current turn . 35 - The < think > must not reveal information that the visible assistant target withholds . 36 - The < think > may use the original assistant target to check stance alignment and response policy , but it must not quote , closely paraphrase , or explicitly pre - announce the target ’ s distinctive surface wording , actions , or inner thoughts . 37 - Treat future - biographical details in the profile as unsafe unless the character could know or act from them at this story point . 38 39 5. Memory decision : 40 - Independently decide whether the scene memory pool is needed . 41 - Scene memory pool means only the listed Relevant Past Memories , not the character profile . 42 - If the current dialogue , current scenario , and profile are sufficient , memory should be marked not needed . 43 - If the target requires information absent from current context / profile but present in a relevant memory , memory should be marked needed . 44 - Check that the generated CoT names the correct memory ids when memory is needed . 45 - When memory is not needed , the CoT should explain why the current dialogue , current scenario , and / or profile are sufficient . 46 - If memory is needed , check that the generated CoT explains how each activated memory affects behavior in the current turn . 47 - If multiple memories are activated , check that the generated CoT compares their directions : aligned , opposed , or mixed , and explains the final net effect . 48 49 6. Profile vs memory : 50 - If the required information is in the profile , the CoT should not claim that a scene memory is needed . 51 - It may say profile knowledge is needed , but scene memory pool is not . 52 53 7. Answer leakage : 54 - The < think > must not quote or closely paraphrase the final assistant wording . 55 - It may describe response function , tone , and constraints , but not the final surface form . 56 57 8. Training usefulness : 58 - The < think > should clearly analyze the current information need , memory relevance , and response policy . 59 - It should not be a generic template or a repetition of the final answer . |
Table 11: A reference prompt for CoT validation.
| Unified End-to-End Behavior-Prediction Prompt. |
| ⬇ 1 You are roleplaying as social - media user { username }. Given the user profile , retrieved pre - event history , current thread , and candidate targets , first select the interaction target . Conditioned on that target , predict the user ’ s recorded engagement as exactly one of Comment , Like , or Repost . Generate textual content only for Comment . Base both decisions on evidence in the input and do not sound like an assistant . 2 3 [ Persona ] 4 { rendered_profile } 5 6 [ Retrieved Past Comments ] 7 { rendered_history_block } 8 9 [ Thread ] 10 [ post ] @ { post_author } -- "{ post_title }" 11 { post_text } 12 13 [ Comments ] 14 - [{ cand_id_1 }] @ { author_1 } ( reply to [ post ]): "{ full_text_1 }" 15 - [{ cand_id_2 }] @ { author_2 } ( reply to [{ parent_id }]): "{ full_text_2 }" 16 17 [ Candidate Targets ] 18 [ post ], [{ cand_id_1 }], [{ cand_id_2 }] 19 20 Rules : 21 - Select a valid target before choosing the action ; neither the gold target nor the gold action is provided . 22 - Conditioned on the selected target , infer exactly one action from { Comment , Like , Repost } and keep the target fixed . 23 - Emit valid JSON for exactly one action and omit all fields belonging to the other actions . 24 - Comment : output action =" Comment " and a reply object containing reply_to_id and natural , thread - specific text . 25 - Like : output action =" LikeThread " and a target object containing id and kind ; LikeThread and LikeBasic are evaluated as high - level Like . A like represents light endorsement , acknowledgment , or agreement . 26 - Repost : output action =" Repost " and the same target object ; a repost represents stronger public amplification than a like . 27 - Do not generate visible text for Like or Repost . Use history only as preference evidence and keep private reasoning concise and grounded . 28 29 Output exactly : 30 < think > 31 - Selected target : ... 32 - Predicted action / intensity for this target : Comment , Like , or Repost / ... 33 - Context grounding : ... 34 - Persona / memory / stance cue : ... 35 - Naturalness cue : ... 36 </ think > 37 38 Then output exactly one valid < answer > branch ; do not output branch labels . 39 Comment : < answer >{" activity_type ":" BehaviorPrediction "," action ":" Comment "," reply ":{" reply_to_id ":"< candidate id >"," text ":"..."}}</ answer > 40 Like : < answer >{" activity_type ":" BehaviorPrediction "," action ":" LikeThread "," target ":{" id ":"< candidate id >"," kind ":" post | comment "}}</ answer > 41 Repost : < answer >{" activity_type ":" BehaviorPrediction "," action ":" Repost "," target ":{" id ":"< candidate id >"," kind ":" post | comment "}}</ answer > |
Table 12: Unified end-to-end behavior-prediction template. All inputs and target-first reasoning steps are shared; only the selected action and its required output fields differ.
| A Reference Prompt for Reference CoT Construction. |
| ⬇ 1 # Action - DPO Reference CoT Prompt 2 3 You are writing inverse reasoning for a persona - simulation training example . 4 5 You are given : 6 - the user ’ s profile and retrieved history , 7 - the current post / thread context , 8 - the ground - truth action and answer block . 9 10 Your job is not to predict a new action . Your job is to explain , briefly and plausibly , why this specific user would take the provided ground - truth action in this situation . 11 12 ## What To Cover 13 14 Write a compact natural - language < think > block . It should cover : 15 - Action fit : why this action type is natural for the user here . 16 - Interaction intensity : why the amount of engagement is calibrated . 17 - Context grounding : what in the current post / thread creates the opening . 18 - Persona and memory : which profile / history signals matter . 19 - Stance and tone : the concrete stance or affective register implied by the action . 20 - Format / target : keep the selected target fixed and include visible target references only when the chosen action format requires them . 21 22 ## Constraints 23 24 - Do not invent private motives , fake memories , or unsupported ideology . 25 - Do not claim that the exact hidden ground - truth text is the only possible wording . 26 - For like / repost actions , do not over - explain as if the user wrote a comment . 27 - Keep the reasoning concise ; this should be a training CoT , not an essay . 28 - Output only one < think >...</ think > block . Do not output an < answer > block . 29 30 ## Output Format 31 32 < think > 33 Action fit : ... 34 Interaction intensity : ... 35 Context grounding : ... 36 Persona and memory : ... 37 Stance and tone : ... 38 Format / target : ... 39 </ think > |
Table 13: A reference prompt for constructing inverse CoT supervision from a known ground-truth action.
| A Reference Prompt for Pairwise DPO Judging. |
| ⬇ 1 You are a strict pairwise preference judge for persona - simulation DPO data . 2 3 You will receive : 4 - the target user ’ s profile and retrieved past comments ; 5 - the current social - media thread and candidate targets ; 6 - the correct high - level action type only : comment / reply , like , or repost ; 7 - Candidate A and Candidate B , each containing CoT , target , action , and optional visible reply text . 8 9 Important : 10 - The ground - truth visible content is hidden . Do not assume exact wording . 11 - LikeThread and LikeBasic are both treated as high - level like . 12 - Judge target grounding first , then the action conditioned on that target , persona fit , memory use , stance , naturalness , specificity , format , and CoT plausibility . 13 - For comment / reply , judge whether the visible text is plausible for this user and context . 14 - For like / repost , penalize unnecessary visible comments . 15 - Do not reward length or polished assistant - like writing by itself . 16 17 Score each candidate from 1 to 5: 18 R1 . Action Fidelity 19 R2 . Interaction Intensity Calibration 20 R3 . Target and Thread Grounding 21 R4 . Persona Consistency 22 R5 . Memory Activation Discipline 23 R6 . Stance Consistency 24 R7 . Response Naturalness 25 R8 . Specificity 26 R9 . Format Validity 27 R10 . CoT Plausibility 28 29 Use the weighted score : 30 weighted_score = (2* R1 + R2 + R3 + R4 + R5 + R6 + R7 + R8 + R9 + R10 ) / 11 31 32 Choose A , B , tie , or drop . 33 Drop weak comparisons if : 34 - confidence < 4; 35 - score gap is too small ; 36 - candidates are near - duplicates ; 37 - the winner wins mainly by being longer ; 38 - either candidate has invalid target , action , format , or malformed JSON ; 39 - the CoT fabricates private psychology or forced memory . 40 41 Return JSON only : 42 { 43 " sample_id ": "...", 44 " correct_action_type ": " comment | like | repost | other ", 45 " candidate_a_scores ": { 46 " R1 ": 1, " R2 ": 1, " R3 ": 1, " R4 ": 1, " R5 ": 1, 47 " R6 ": 1, " R7 ": 1, " R8 ": 1, " R9 ": 1, " R10 ": 1, 48 " weighted_score ": 0.0 49 }, 50 " candidate_b_scores ": { 51 " R1 ": 1, " R2 ": 1, " R3 ": 1, " R4 ": 1, " R5 ": 1, 52 " R6 ": 1, " R7 ": 1, " R8 ": 1, " R9 ": 1, " R10 ": 1, 53 " weighted_score ": 0.0 54 }, 55 " chosen ": " A | B | tie | drop ", 56 " confidence ": 1, 57 " score_gap ": 0.0, 58 " winner_reason ": " brief evidence - bound reason ", 59 " loser_weakness ": " main concrete weakness ", 60 " keep_for_dpo ": true 61 } |
Table 14: A reference prompt for pairwise DPO preference judging.
| A Reference Prompt for Seven-Dimension Semantic Evaluation. |
| ⬇ 1 You are a strict , consistent , and objective evaluator for social media user simulation . 2 Your task is to assess how well the model - generated comment simulates the target user ’ s real response in the same social media context . 3 You will be given : 4 1. The original social media post 5 2. The target user ’ s history , profile , or memory information 6 3. The target user ’ s real comment , used as the ground truth 7 4. The model - generated comment 8 Important evaluation principles : 9 - The ground - truth comment is not the only possible correct response , but it represents the user ’ s actual reaction in this context . 10 - Do not judge only by lexical overlap . Evaluate semantic meaning , interaction intent , emotion , stance , style , persona / memory consistency , and naturalness . 11 - If the generated comment uses different wording but expresses a similar meaning , attitude , emotion , and interaction goal , it can receive a high score . 12 - If the generated comment is semantically relevant but does not match the user ’ s style , stance , or interaction intent , reduce the corresponding scores . 13 - If the generated comment fabricates the user ’ s personal history , identity , preferences , relationships , or experiences , significantly reduce the persona_memory_consistency score . 14 - If the generated comment sounds like AI writing , is overly formal , overly explanatory , or resembles a summary , reduce naturalness and style_fidelity scores . 15 - Do not give a high score simply because the generated comment is more complete or grammatically fluent . The goal is whether it sounds like the target user . 16 - Be strict . A score above 90 should only be given when the generated comment is extremely close to the target user ’ s real response . 17 - Avoid giving all dimensions similar scores . Evaluate each dimension independently . 18 [ Input ] 19 Original social media post : { source_post } 20 Target user history / profile / memory : { user_profile_or_history } 21 Ground - truth comment : { ground_truth_comment } 22 Model - generated comment : { generated_comment } 23 [ Scoring Scale ] Assign an integer score from 0 to 100 per dimension . 24 90-100: Nearly perfect match . 25 75-89: Strong match , only minor differences . 26 60-74: Moderate match , noticeable differences . 27 40-59: Weak match , substantial deviation . 28 20-39: Poor match . 29 0-19: Very poor ; irrelevant , contradictory , or fabricated . 30 [ Evaluation Dimensions ] 31 1. semantic_alignment 32 Assess whether the generated comment expresses the same core meaning , topic , and information points as the ground - truth comment . 33 2. intent_alignment 34 Assess whether the generated comment fulfills the same social interaction purpose ( e . g ., supporting , opposing , joking , asking , answering , expressing emotion , sharing experience , giving advice ). 35 3. emotion_consistency 36 Assess whether the generated comment matches the ground - truth in emotion type , direction , and intensity . 37 4. stance_consistency 38 Assess whether the generated comment expresses the same stance ( supportive , opposed , skeptical , neutral , ironic , etc .) toward the post , event , or opinion . 39 5. style_fidelity 40 Assess whether the generated comment resembles the target user ’ s language style : length , tone , punctuation , emoji usage , slang , sentence patterns , formality , and humor style . 41 6. persona_memory_consistency 42 Assess whether the generated comment is consistent with the target user ’ s historical preferences , interests , values , and long - term behavioral tendencies , and avoids fabricating user facts . 43 7. naturalness 44 Assess whether the generated comment sounds like a real social media comment rather than AI - generated text , a summary , or an essay - like response . 45 [ Overall Score ] 46 overall_score = unweighted average of the seven dimension scores , rounded to two decimal places . 47 [ Output ] JSON only . No markdown . No explanatory preface . All dimension scores must be integers from 0 to 100. 48 { 49 " semantic_alignment ": {" score ": 0, " reason ": ""}, 50 " intent_alignment ": {" score ": 0, " reason ": ""}, 51 " emotion_consistency ": {" score ": 0, " reason ": ""}, 52 " stance_consistency ": {" score ": 0, " reason ": ""}, 53 " style_fidelity ": {" score ": 0, " reason ": ""}, 54 " persona_memory_consistency ": {" score ": 0, " reason ": ""}, 55 " naturalness ": {" score ": 0, " reason ": ""}, 56 " overall_score ": 0.00, 57 " summary ": "" 58 } |
Table 15: A reference prompt for seven-dimension semantic evaluation.
More in this topic
Human Behavior Simulation
Human behavior simulation models how individuals and groups decide, interact, move, and adapt in social, economic, educational, and physical settings.
See all 240 papers