CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

Yixin Nie; Lin Guan; Zhongyao Ma; Anchit Gupta; Yipin Zhou; Xiao Li; Zhengping Zhou; Raymond Zeng

cs.CLMar 2, 2026

CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

Authors: Yixin Nie, Lin Guan, Zhongyao Ma, Anchit Gupta, Yipin Zhou, Xiao Li, Zhengping Zhou, Raymond Zeng,

Organizations: Meta Superintelligence Labs · OpenAI · xAI · FAIR at Meta

Abstract

This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1, we refined models across 15 generations using data from both internal and external real-user traffic. Through continuous deployments from July 2024 to April 2025, we conducted controlled 7-day A/B tests showing consistent engagement improvements: 7 of 8 newly deployed models demonstrated positive lift over the baseline, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth. We also observed substantial gains in steerability, with instruction following increasing from 59.2% to 84.8% and instruction violations decreasing from 26.6% to 5.8%. We detail the CharacterFlywheel process which integrates data curation, reward modeling to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step. We also discuss our methods for overfitting prevention and navigating production dynamics at scale. These contributions advance the scientific rigor and understanding of LLMs in social applications serving millions of users.

Figures & tables

Panel of Figure 2 — Figure 2: Landscape Climbing

Figure 7: The CharacterFlywheel Iterative Development Cycle.

Figure 8: The Data Pipeline.

Constraint/Dimension	Definition	Rationale
First turn conversation ratio	Ensure at least 10% of samples are First turn conversations	Improve initial response quality, which is critical for first impressions
Per-character cap	Limit conversations per character to maximum 3%, regardless of popularity	Promote balanced learning across personas and prevent over-representation
Locale and language	Geographic region and language of user interactions	Ensure diverse and representative coverage across user populations
Job-to-be-done (JTBD)	User’s underlying intention or goal behind a prompt	Align model capabilities with actual user needs and use cases
Conversation depth	Interaction length, ranging from short-term (welcome messages, first impressions) to long-term (sustained engagement)	Balance optimization for both initial user experience and power user retention
Prompt quality	Measure of how well the LLM understands and responds to the prompt	Prioritize high-quality interactions that yield clearer training signals

Table 1: Curation constraints and monitoring dimensions

Panel of Figure 9 — Figure 9: Popular Publicly Shared Characters.

Signal	Description	Rationale
$p(\text{continue})$	User continues the conversation after receiving the model response within 10 minutes	Indicates engaging and satisfactory content that prompts further interaction
$p(\text{love})$	User gives a ”love” emoji reaction to the last model response	Strong positive feedback indicating high-quality, emotionally resonant response
$p(\text{thumb up})$	User gives a ”thumbs up” emoji reaction to the last model response	Positive feedback indicating helpful or accurate response
$p(\text{thumb down})$	User gives a ”thumbs down” emoji reaction to the last model response	Negative feedback indicating poor quality, inaccurate, or unhelpful response
$p(\text{feedback})$	User provides explicit written feedback on the last model response	Indicates response needs improvement; provides detailed signal for model refinement

Table 2: User signals experimented.

r_{\text{max}}=\max_{j=1,...,k}r(X_{i},Y_{i,j}),\ \ \ j^{*}=\argmax_{j=1,...,k}r(X_{i},Y_{i,j})

Algorithm 1 Rejection Sampling

General	MMLU ( Hendrycks et al., 2021a ) , IFEval ( Zhou et al., 2023 )
Reasoning	ARC-Challenge ( Clark et al., 2018 ) , GPQA ( Rein et al., 2024 ) , HellaSwag ( Zellers et al., 2019 )
Math	GSM8K ( Cobbe et al., 2021 ) , Math ( Hendrycks et al., 2021b )
Code	HumanEval ( Chen et al., 2021 ) , MBPP ( Austin et al., 2021 )

Table 3: Community benchmarks used for offline evaluation.

Metric Name	Prompt Set	Type	Measurement	Rationale
Avg. Response Length	User	Rule-based	Avg. token count	Ensures responses are appropriately detailed without being overly verbose
Contains List	Internal	Rule-based	% of responses with lists	Evaluates structured formatting for clarity and organization
Contains Emoji	Internal	Rule-based	% of responses with emojis	Monitors appropriate use of emojis for tone and character alignment
Positive Sentiment	Internal	LLM-as-a-judge	% of responses with positive sentiment	Assesses overall emotional tone and helpfulness
Instruction Violation	Internal	LLM-as-a-judge	% of responses violating user instructions	Evaluates character steerability and instruction compliance
Cooperative Ratio	Internal	LLM-as-a-judge	% of responses that are cooperative	Measures willingness to assist vs. unnecessary refusals

Table 4: Response Characteristics Metrics

Figure 11: Pre-launch quality progression (V2-V7). Left panel: Human win rates comparing CharacterFlywheel models against GPT-4o baseline, showing improvement from 37.4% (V3) to 46.2% (V7). Right panel: Win rates against the immediate previous model version, displaying both human evaluations (blue bars, 50.2%-52.5%) and reward model predictions (orange bars, 53.6%-57.6%). Both signals consistently exceed the 50% neutral threshold (dashed line), indicating quality improvements across iterations.

Figure 12: Pre-launch engagement validation. Engagement breadth and depth metric lifts with 95% confidence intervals (Section 2.5.2 ) for V2-V4, tested on a fixed set of characters with random online users.

Figure 13: Post-launch engagement trajectory (V8-V15). Top panel: A/B test engagement lifts comparing each version against the production baseline. Green markers indicate statistically significant positive results, red indicates significant negative, and gray indicates non-significant. Middle panel: Reward model win rates over time for models trained on internal traffic (blue) and user traffic (orange). The spike at V12, where RM User exceeds 70% while online engagement degrades, illustrates reward overfitting. Bottom panel: Cumulative Engagement growth (baseline offset to 1) showing sustained upward trend of engagement though not entirely attributable to model updates.

Category	Benchmark	Llama 3 8B	Llama 3 70B	CharacterFlywheel V7	Llama 3 405B	GPT-4o	Claude 3.5 Sonnet
General	MMLU (0-shot)	60.4	83.6	79.5	87.3	89.1	89.9
	IFEval	84.4	87.5	84.8	88.6	85.6	88.0
Code	HumanEval (0-shot)	72.6	80.5	77.4	89.0	90.2	92.0
	MBPP (3-shot CoT)	71.7	86.0	66.6	88.6	87.8	90.5
Math	GSM8K (8-shot CoT)	84.5	95.1	92.3	96.8	96.1	96.4
	MATH (0-shot CoT)	51.9	68.0	50.5	73.8	76.6	71.1

Table 5: Benchmark comparison. CharacterFlywheel V7 performance compared to Llama 3 family and leading commercial models. CharacterFlywheel V7 is fine-tuned from Llama 3 70B (highlighted baseline).

Figure 14: Benchmark progression across pre-launch versions (V2-V7). Performance across nine standard benchmarks shows general stability throughout development, with most metrics converging by V6-V7. Notable improvements occur in IFEval and ARC Challenge, while MBPP shows some degradation, reflecting the trade-offs in optimizing for conversational engagement.

Version	V2	V3	V4	V5	V6	V7	V8
Instruction Violation (%)	26.6	22.6	17.9	22.2	13.7	7.5	5.8

Table 6: Character steerability improvement. Instruction violation rates measured via LLM-as-a-judge on interactive chat.

Figure 15: Response characteristics across model versions (V1-V15). Evolution of false refusal metrics (top row), formatting patterns (second row), tone and sentiment (third row), and quality indicators (bottom row). Percentage labels indicate relative change from V1 to V15. Metrics are evaluated using LLM-as-a-judge on internal annotated traffic and rule-based methods on user traffic evaluation sets.

Failures	Description	Mitigation strategies
Infinity Loop	The model occasionally generates repetitive responses, resulting in excessively long outputs that exceed the token limit.	Employed token count as a heuristic to identify prompts that trigger repetitive responses; Masked EOT tokens in DPO and online DPO loss computation; Introduced a long response penalty in online RL. Led to a 57.8% lower loop rate in online traffic.
Repetitive Phrases	The model over-uses common phrases, such as ”OMG” (4.14%), ”LOL” (3.3%), ”mind blown” (3.2%), and templated refusal. Moreover, markdown formatting in responses is inconsistent (e.g., 37% unclosed or broken formatting).	Verified that the model has a tendency to follow undesirable “biases” in prompts (Sec. 3.5.6 ). Developed a linter to remove or reduce the prevalence of repetitive phrases and to enforce consistent Markdown formatting in SFT data and online RL prompts. Reduced repetitive phrases by 60% in online traffic.
Hindi Language Mismatches	The model responds in English to users who chat in Hindi and sometimes provides unrequested translations to English.	Curated SFT and DPO datasets based on failures in online traffic; Collected extra preference pairs to correct the RM’s bias towards English responses in Hindi prompts. Reduced LMR failures from 75% to 1.5%, and the prevalence of implicit translation from 64.4% to 1.1% on a hard evaluation set.

Table 7: Failure modes and mitigation strategies.

	Static Chat Preference Data					Interactive Chat Preference Data
Model	Batch	Batch	Batch	Batch	All Data	Batch	Batch	Batch	Batch	All Data
	240923	241018	241118	241229	until 241229	240923	241018	241118	241229	until 241229
RM_240923	0.725	0.572	0.558	0.575	0.652	0.608	0.503	0.500	0.513	0.573
RM_241018	0.715	0.721	0.557	0.555	0.672	0.634	0.644	0.510	0.521	0.627
RM_241118	0.732	0.743	0.725	0.619	0.720	0.645	0.624	0.628	0.546	0.632
RM_241229	0.742	0.735	0.754	0.742	0.746	0.644	0.631	0.648	0.669	0.650

Table 8: Pointwise Preference Model Performance Across Different Data Batches.

Training Data	Pointwise RM Acc (%)		Pairwise RM Acc (%)
	Evaluation Set		Evaluation Set
	Single (Rand)	Multi (Agree)	Single (Rand)	Multi (Agree)
Baseline (Not Trained)	59.01	66.84	59.10	67.08
Multi-Review (With Agreement)	59.02	70.86	60.60	72.87
Single-Review (All)	58.94	70.59	61.50	72.08
Single-Review (Random)	60.18	70.96	60.70	72.50

Table 9: Impact of annotation agreement on engagement preference accuracy. We compare a baseline (Not Trained) against models trained on different data variants, using two distinct evaluation sets.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

User Engagement Metrics
Metric	V1	V2	V3	V4	V5	V6	V7	V8	V9	V10	V11	V12	V13	V14	V15
Release Date	Mar 11	Mar 28	Apr 23	May 28	Jun 17	Jul 10	Jul 22	Aug 19	Sep 11	Sep 30	Oct 25	Nov 25	Jan 5	Feb 14	Mar 25
Engagement Breadth Lift (1P) (%)	–	$0.84^{+0.74}_{-0.74}$	$0.40^{+0.74}_{-0.74}$	$0.62^{+1.1}_{-1.1}$	–	–	–	–	–	–	–	–	–	–	–
Engagement Depth Lift (1P) (%)	–	$3.2^{+2.7}_{-2.7}$	$2.6^{+2.5}_{-2.4}$	$2.1^{+3.9}_{-3.8}$	–	–	–	–	–	–	–	–	–	–	–
Engagement Breadth Lift (AI Studio) (%)	–	–	–	–	–	–	–	$-0.38^{+1.33}_{-1.32}$	$1.73^{+1.03}_{-1.02}$	$2.09^{+1.16}_{-1.15}$	$4.47^{+1.01}_{-1.00}$	$0.05^{+0.43}_{-0.45}$	$2.12^{+0.56}_{-0.56}$	$8.8^{+2.4}_{-2.3}$	$0.41^{+0.53}_{-0.53}$
Engagement Depth Lift (AI Studio) (%)	–	–	–	–	–	–	–	$3.52^{+4.47}_{-4.29}$	$1.38^{+4.62}_{-4.41}$	$3.8^{+3.5}_{-3.4}$	$18.2^{+2.5}_{-2.4}$	$-2.9^{+2.7}_{-2.7}$	$19.4^{+2.3}_{-2.2}$	$11.2^{+4.0}_{-3.8}$	$5.0^{+1.0}_{-1.0}$

Appendix

Table 10: Performance metrics across model versions

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Language Models Learn to Improve Language Models
Oct 16, 2026Minhua Lin, Zijun Wang, Zewen Liu +7Language Model Self-Improvement Large Language Model-Guided Optimization
The Pennsylvania State University · UC Santa Cruz · Emory University +2
Chain-of-Experience for Continual LLM Improvement
Aug 18, 2026Haoqin Tu, Yunhao Fang, Yizhong Wang +2Language Model Self-Improvement Continual Learning for LLMs
UC Santa Cruz · Bytedance Seed
Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction
Feb 26, 2026Nils Schwager, Simon Münker, Alistair Plum +1Large Language Model-Based Social Simulation Language Model Generation Evaluation
Trier University, Trier, Germany · University of Luxembourg, Esch-sur-Alzette, Luxembourg

Oct 16, 2026J/K move · Enter open · S save

Language Models Learn to Improve Language Models

Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng+6

The Pennsylvania State University · UC Santa Cruz · Emory University+2

Language Model Self-Improvement Large Language Model-Guided Optimization RL for Language Models Reward Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

Abstract

LLM Post-Training

Explore similar work

Language Models Learn to Improve Language Models

Chain-of-Experience for Continual LLM Improvement

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction