Organizations: Monash University · Harbin Institute of Technology, Shenzhen · Hainan University · South China University of Technology · The Chinese University of Hong Kong, Shenzhen
Self-reflection is a basic self-improvement skill of language agents, yet it does not reliably improve their outputs, where the underlying behavioral mechanism remains opaque. We introduce ReBeCA (self-Reflection Behavior explained through Causality-inspired Analysis), a framework for identifying and validating outcome-relevant reflective behaviors from recorded trajectories. ReBeCA uses reflective codes, an inductive coding schema, to represent each trajectory as a sequence of reflective behaviors. It learns dependencies among these behaviors and the outcome with causal discovery, selects the behaviors that best predict the outcome of held-out trajectories, and tests them for consistency across data subsets and by prompt-based steering. In a controlled case study of four Qwen3 models under fixed-round Self-Refine on mathematical reasoning and translation, ReBeCA identifies compact, round-specific behavioral sets with superior held-out predictive fit and uses prompt-based steering on unseen problems to reveal how individual and combined behavioral directives affect performance (Holm-adjusted p=.020 on Qwen3-4B). ReBeCA offers a reusable framework that turns agent trajectories into interpretable, verifiable behavioral hypotheses without requiring model internals or preselecting the behaviors to test.
Figures & tables
Figure 1: Overview of ReBeCA. (1) Self-Refine produces trajectories; (2) reflective codes record which behaviors occur at each round, giving the binary matrix S ; (3) causal discovery learns a dependency graph G^ over the coded behaviors and y^ , and the parents of y^ (shaded) are retained as behavioral hypotheses and tested on held-out trajectories and by prompt-based steering.
Figure 2: Illustrative spans for Clarity and Organization. A reflective code pairs a concept with a coding description that specifies which text spans express it; each code is recorded at each round as present or absent.
Figure 3: ReBeCA as a discovery pipeline. Coded trajectories yield a hypothesized parent set P^ of y^ , which is assessed through C1–C3 (grey) on both tasks and steering on unseen problems C4 (blue) for MATH.
Figure 4: Consistency-Enhanced Self-Refine (CESR). At each step, K=20 samples are embedded and clustered, and the medoid ( ★ ) of the largest cluster becomes the step output, so that each step reflects the model’s typical output rather than a single sample.
Figure 5: Dependency structures learned by ReBeCA across 4F GES runs on both tasks. Edges shown appear in at least half of the runs (labels: run counts; 28 MATH runs, 20 per BOUQuET metric). This recurrence summary differs from CVLL-based set selection in Table 1 .
Table 1: Stages 1–2: candidate parent sets of y^ from GES across folds and scores, ranked by CVLL, i.e., how well each set predicts the outcome of held-out trajectories (mean CVLL across folds ± SD of individual held-out log probabilities). Bold marks the highest mean CVLL among candidates; selected sets also outperform the null and full models.
(a) MATH: effect size and consistency (Stage 3)
Fold-group test ( p )
Coefficient [95% CI]
Mean
Variance
ηp2 ↓
CO2
.298 [ .259 , .338 ]
.956
.950
.019
Sp4
.160 [ .122 , .198 ]
.860
.317
.045
Intercept
.514 [ .488 , .540 ]
.942
.571
.024
Table 2: Dependence on the selected parents. (a) MATH: linear probability coefficients (difference in success rate; mean and 95% CI over M=20 folds), compared across four fold groups in mean (ANOVA) and variance (Levene), with between-group effect sizes ηp2 . (b) BOUQuET: final versus initial translation under each metric; Equal denotes equal metric scores; with CO3 coded, both Better and Worse outcomes are more frequent.
Reference
Round 2
Round 3
Please don’t look at me like this.
Please don’t stare at me or look at me with disapproval.
Please don’t look at me that way.
COMET ↑ .525
COMET ↑ .784
Feeling guilty?
Do you feel guilty?
Have you felt guilty lately?
COMET ↑ .417
COMET ↑ .335
Table 3: Two round-3 revisions coded as Clarity and Organization (Qwen3-8B on BOUQuET; Burmese and Greek sources): one removes content added in round 2 and raises COMET, the other adds content absent from the source and lowers it.
Correct answers ↑
Steering
0.6B
1.7B
4B
8B
Total
None
1
13
21
24
59
CO2
1
14
26
23
64
Sp4
1
10
33
18
62
CO2 + Sp4
0
9
19
21
49
Table 4: Steering either code alone raises correct solutions (59 → 64 or 62), whereas joint steering drops total to 49. Success rates differ significantly on Qwen3-4B (Holm-adjusted p=.020 , Cochran’s Q).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Self-Refine prompts, reproduced verbatim. The query x is the first user turn. The feedback prompt follows the latest response and the refinement prompt follows the generated feedback, both as user turns and identically at every round.
Code
Dimension addressed by the feedback or the revision
Shared by MATH and BOUQuET (MT)
Completeness
coverage of all parts of the query or source text, e.g., supplying omitted content
Specificity (Sp)
specificity of the content, e.g., replacing broad statements with specific ones
Clarity and Organization (CO)
structure and readability of the response, e.g., reordering or clarifying steps
Depth of Analysis (DA)
reasoning beyond the surface level, e.g., explaining connections between steps
Appendix
Table 5: Reflective codebook. All four codes are shared by MATH and BOUQuET (MT).
Figure 7: Annotation prompts for the code Completeness. Prompts for the other codes use the same template and differ in the target code. {query}, {ground_truth}, {prev_resp}, {feedback}, and {refined_resp} hold the query, the reference output, yt−1 , fbt−1 , and yt of the annotated round.