Organizations: Monash University · Harbin Institute of Technology, Shenzhen · Hainan University · South China University of Technology · The Chinese University of Hong Kong, Shenzhen
Self-reflection is a basic self-improvement skill of language agents, yet it does not reliably improve their outputs, where the underlying behavioral mechanism remains opaque. We introduce ReBeCA (self-Reflection Behavior explained through Causality-inspired Analysis), a framework for identifying and validating outcome-relevant reflective behaviors from recorded trajectories. ReBeCA uses reflective codes, an inductive coding schema, to represent each trajectory as a sequence of reflective behaviors. It learns dependencies among these behaviors and the outcome with causal discovery, selects the behaviors that best predict the outcome of held-out trajectories, and tests them for consistency across data subsets and by prompt-based steering. In a controlled case study of four Qwen3 models under fixed-round Self-Refine on mathematical reasoning and translation, ReBeCA identifies compact, round-specific behavioral sets with superior held-out predictive fit and uses prompt-based steering on unseen problems to reveal how individual and combined behavioral directives affect performance (Holm-adjusted p=.020 on Qwen3-4B). ReBeCA offers a reusable framework that turns agent trajectories into interpretable, verifiable behavioral hypotheses without requiring model internals or preselecting the behaviors to test.
Figures & tables
Figure 1: Overview of ReBeCA. (1) Self-Refine produces trajectories; (2) reflective codes record which behaviors occur at each round, giving the binary matrix S ; (3) causal discovery learns a dependency graph G^ over the coded behaviors and y^ , and the parents of y^ (shaded) are retained as behavioral hypotheses and tested on held-out trajectories and by prompt-based steering.
Figure 2: Illustrative spans for Clarity and Organization. A reflective code pairs a concept with a coding description that specifies which text spans express it; each code is recorded at each round as present or absent.
Figure 3: ReBeCA as a discovery pipeline. Coded trajectories yield a hypothesized parent set P^ of y^ , which is assessed through C1–C3 (grey) on both tasks and steering on unseen problems C4 (blue) for MATH.
Figure 4: Consistency-Enhanced Self-Refine (CESR). At each step, K=20 samples are embedded and clustered, and the medoid ( ★ ) of the largest cluster becomes the step output, so that each step reflects the model’s typical output rather than a single sample.
Figure 5: Dependency structures learned by ReBeCA across 4F GES runs on both tasks. Edges shown appear in at least half of the runs (labels: run counts; 28 MATH runs, 20 per BOUQuET metric). This recurrence summary differs from CVLL-based set selection in Table 1 .
Table 1: Stages 1–2: candidate parent sets of y^ from GES across folds and scores, ranked by CVLL, i.e., how well each set predicts the outcome of held-out trajectories (mean CVLL across folds ± SD of individual held-out log probabilities). Bold marks the highest mean CVLL among candidates; selected sets also outperform the null and full models.
(a) MATH: effect size and consistency (Stage 3)
Fold-group test ( p )
Coefficient [95% CI]
Mean
Variance
ηp2 ↓
CO2
.298 [ .259 , .338 ]
.956
.950
.019
Sp4
.160 [ .122 , .198 ]
.860
.317
.045
Intercept
.514 [ .488 , .540 ]
.942
.571
.024
Table 2: Dependence on the selected parents. (a) MATH: linear probability coefficients (difference in success rate; mean and 95% CI over M=20 folds), compared across four fold groups in mean (ANOVA) and variance (Levene), with between-group effect sizes ηp2 . (b) BOUQuET: final versus initial translation under each metric; Equal denotes equal metric scores; with CO3 coded, both Better and Worse outcomes are more frequent.
Reference
Round 2
Round 3
Please don’t look at me like this.
Please don’t stare at me or look at me with disapproval.
Please don’t look at me that way.
COMET ↑ .525
COMET ↑ .784
Feeling guilty?
Do you feel guilty?
Have you felt guilty lately?
COMET ↑ .417
COMET ↑ .335
Table 3: Two round-3 revisions coded as Clarity and Organization (Qwen3-8B on BOUQuET; Burmese and Greek sources): one removes content added in round 2 and raises COMET, the other adds content absent from the source and lowers it.
Correct answers ↑
Steering
0.6B
1.7B
4B
8B
Total
None
1
13
21
24
59
CO2
1
14
26
23
64
Sp4
1
10
33
18
62
CO2 + Sp4
0
9
19
21
49
Table 4: Steering either code alone raises correct solutions (59 → 64 or 62), whereas joint steering drops total to 49. Success rates differ significantly on Qwen3-4B (Holm-adjusted p=.020 , Cochran’s Q).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Self-Refine prompts, reproduced verbatim. The query x is the first user turn. The feedback prompt follows the latest response and the refinement prompt follows the generated feedback, both as user turns and identically at every round.
Code
Dimension addressed by the feedback or the revision
Shared by MATH and BOUQuET (MT)
Completeness
coverage of all parts of the query or source text, e.g., supplying omitted content
Specificity (Sp)
specificity of the content, e.g., replacing broad statements with specific ones
Clarity and Organization (CO)
structure and readability of the response, e.g., reordering or clarifying steps
Depth of Analysis (DA)
reasoning beyond the surface level, e.g., explaining connections between steps
Appendix
Table 5: Reflective codebook. All four codes are shared by MATH and BOUQuET (MT).
Figure 7: Annotation prompts for the code Completeness. Prompts for the other codes use the same template and differ in the target code. {query}, {ground_truth}, {prev_resp}, {feedback}, and {refined_resp} hold the query, the reference output, yt−1 , fbt−1 , and yt of the annotated round.
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
Siqi Zeng, Andre N. Assis, Rowan Wang
University of Illinois · Constellation · Anthropic
Reasoning language models that can call tools must decide during inference whether to answer unaided or delegate. Any self-reflection mechanism for this must answer three questions: where the reflective signal comes from (verbal reports, output distributions, hidden states, a separate predictor), how it is presented to the model (numerical prediction, confidence token, prompt injection), and whether it changes the model's subsequent action. We isolate the third question. At a fixed point in otherwise identical reasoning trajectories, we insert a single first-person sentence expressing either confidence or doubt; the model then continues reasoning and chooses whether to answer directly or call a tool. Comparing these counterfactual continuations measures the causal effect of the reflective signal on delegation. We call this behavioral response Nudgeability and measure it along two dimensions: sensitivity, how strongly confidence and doubt change delegation rates, and targeting, whether delegation increases for problems the model cannot solve unaided and decreases for those it can. Across nine small-to-medium open-weight reasoning models from three families (Qwen, Gemma, and GLM) and two tasks, models are consistently sensitive: doubt increases delegation and confidence decreases it, with a median confidence-to-doubt swing of 20.6 percentage points, and 53 to 70 points for the larger provider-served models. This responsiveness is poorly targeted: a median 42% of induced flips are well-targeted, only a +2 percentage-point lift over a random-selection baseline. Confidence language is thus a strong control surface for delegation, but current models use it only weakly in accordance with their actual competence. Nudgeability offers a simple, post-training-free way to evaluate both sensitivity and targeting as endogenous self-reflection mechanisms mature.
Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from R2=0.24 to R2=0.64 on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from 36% to 92%. In the other direction, behavior updates improve alignment, reducing HarmBench failure rate from 15.0% to 0.5% without substantially increasing refusal on harmless prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.