CoLFA: Collaboration-Guided Lightweight Failure Attribution for LLM-Based Multi-Agent Systems
Organizations: Nanjing University
Abstract
Failure attribution in multi-agent systems requires tracing error formation to identify the decisive error step and the responsible agent. However, agents' division of work and information exchange lead to Interleaved Steps and Cluttered Content, requiring models to link executions and check details in lengthy records. This makes it difficult to balance localization accuracy and token cost. We find that multi-agent collaboration also leaves records of how executions connect and content passes between them as results are formed, providing evidence for tracing error formation. Based on this observation, we propose CoLFA (Collaboration-Guided Lightweight Failure Attribution), which uses collaboration cues to organize the attribution input according to how results are formed, determining which steps are examined and how their content is presented. Through Handoff-Guided Step Expansion and Cross-Referenced Content Organization, CoLFA helps the model focus on examining error formation and the role of each decision while reducing the processing of irrelevant and repeated records. Across five settings spanning Who&When and LongRCA and three backbone models, CoLFA achieves the highest exact step-localization accuracy with the lowest token consumption. Compared with individual baselines in each setting, CoLFA improves accuracy by up to 37.50% and reduces token consumption by up to 97.6%.
Figures & tables
| DeepSeek-V4.1-Flash | GLM-5.3-Flash | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Alg. | Hand. | Overall | Cost (K) | Alg. | Hand. | Overall | Cost (K) | |||||||||
| Method | Agent | Step | Agent | Step | Agent | Step | Tok. | CoP | Agent | Step | Agent | Step | Agent | Step | Tok. | CoP |
| All-at-Once | 60.32 | 23.81 | 60.34 | 3.45 | 60.33 | 17.39 | 8.4 | 48 | 65.08 | 12.70 | 53.45 | 3.45 | 61.41 | 9.78 | 8.2 | 84 |
| A2P | 62.70 | 45.24 | 56.90 | 20.69 | 60.87 | 37.50 | 8.3 | 22 | 64.29 | 50.00 | 56.90 | 20.69 | 61.96 | 40.76 | 8.4 | 21 |
| CORRECT | 61.11 | 50.00 | 48.28 | 22.41 | 57.07 | 41.30 | 22.7 | 55 | 66.67 | 53.17 | 48.28 | 25.86 | 60.87 | 44.57 | 20.6 | 46 |
| Step-by-Step | 42.06 | 32.54 | 51.72 | 20.69 | 45.11 | 28.80 | 111.1 | 386 | 46.03 | 35.71 | 50.00 | 18.97 | 47.28 | 30.43 | 32.0 | 105 |
| DeepSeek-V4.1-Flash | GLM-5.3-Flash | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Role | Exact | 5 | MAE | Tok. (K) | CoP (K) | Role | Exact | 5 | MAE | Tok. (K) | CoP (K) |
| All-at-Once | 42.02 | 1.40 | 10.79 | 77.09 | 52.8 | 3,763 | 37.81 | 1.05 | 11.84 | 60.06 | 52.3 | 4,969 |
| A2P | 46.49 | 13.51 | 29.47 | 43.07 | 53.6 | 397 | 47.02 | 16.49 | 28.16 | 43.65 | 52.9 | 321 |
| CORRECT | 49.65 | 18.86 | 25.70 | 41.88 | 114.5 | 607 | 49.21 | 20.35 | 32.19 | 40.04 | 110.3 | 542 |
| Step-by-Step | 33.42 | 7.63 | 17.37 | 46.16 | 950.0 | 12,449 | 30.61 | 8.33 | 16.67 | 54.62 | 367.8 | 4,414 |
| Binary Search | 15.96 | 0.00 | 15.96 | 77.79 | 130.4 | — | 28.51 | 3.51 | 18.07 | 53.47 | 114.7 | 3,268 |
| Alg. | Hand. | Overall | Cost (K) | |||||
|---|---|---|---|---|---|---|---|---|
| Method | Agent | Step | Agent | Step | Agent | Step | Tok. | CoP |
| All-at-Once | 67.46 | 39.68 | 62.07 | 3.45 | 65.76 | 28.26 | 12.4 | 44 |
| A2P | 65.87 | 50.00 | 67.24 | 29.31 | 66.30 | 43.48 | 13.3 | 31 |
| CORRECT | 65.87 | 47.62 | 67.24 | 32.76 | 66.30 | 42.93 | 33.6 | 78 |
| RAFFLES | 65.08 | 47.62 | 62.07 | 36.21 | 64.13 | 44.02 | 53.5 | 122 |
| DeepDebug | 62.70 | 47.62 | 44.83 | 20.69 | 57.07 | 39.13 | 21.0 | 54 |
| Who&When / DeepSeek | Who&When / GLM | LongRCA / DeepSeek | LongRCA / GLM | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Variant | Agent | Step | Tok. | CoP | Agent | Step | Tok. | CoP | Role | Exact | Tok. | CoP | Role | Exact | Tok. | CoP |
| CoLFA | 64.67 | 44.57 | 1.00 | 1.00 | 66.30 | 46.74 | 1.00 | 1.00 | 51.58 | 21.84 | 1.00 | 1.00 | 52.28 | 23.60 | 1.00 | 1.00 |
| w/o Expansion | 59.78 | 39.13 | 1.10 | 1.25 | 61.96 | 42.39 | 1.09 | 1.21 | 47.02 | 19.65 | 1.47 | 1.63 | 49.82 | 20.70 | 1.47 | 1.68 |
| w/o Organization | 60.87 | 40.22 | 1.09 | 1.20 | 62.50 | 43.48 | 1.08 | 1.17 | 49.47 | 19.30 | 1.35 | 1.52 | 48.07 | 20.35 | 1.36 | 1.57 |
| w/o Both | 57.07 | 36.96 | 1.21 | 1.46 | 57.61 | 39.13 | 1.21 | 1.44 | 45.26 | 17.89 | 2.32 | 2.83 | 46.32 | 19.30 | 2.32 | 2.83 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Who&When | LongRCA |
|---|---|---|
| Execution links and content references | ||
| Specificity threshold | 1.4 | |
| Minimum length of a region replaced by a content reference (words) | 12 | |
| Detailed records (characters) | ||
| Reasoning truncation length | 1,500 | 500 |
| Tool output or environment feedback truncation length | 6,000 | 1,500 |
| Expansion scope | Coverage | Mean step share |
|---|---|---|
| All decision steps of the output and upstream agents, supplemented with directly linked steps | 87.4 | 71.9 |
| Output steps, upstream production steps, and their supply steps (CoLFA) | 78.9 | 49.3 |
| All decision steps of the output agent, supplemented with directly linked steps | 67.2 | 40.7 |
| All decision steps of the upstream agent, backward tracing within the output agent through content correspondence, and directly linked steps | 61.3 | 52.3 |
| Backward tracing within both the output and upstream agents through content correspondence and assignments, supplemented with directly linked steps | 47.5 | 26.1 |
| Traces | Steps | Tokens (K) | |
|---|---|---|---|
| Who&When | |||
| Alg. | 126 | 8.7 | 3.0 |
| Hand. | 58 | 51.6 | 16.9 |
| LongRCA | |||
| TravelPlanner | 685 | 165.9 | 48.4 |
| WebArena Verified | 177 | 58.8 | 31.0 |
| DeepSeek-V4.1-Flash | GLM-5.3-Flash | |||||
| Method | Input | Output | Total | Input | Output | Total |
| Who&When | ||||||
| All-at-Once | 7,881 | 490 | 8,372 | 7,829 | 350 | 8,179 |
| A2P | 8,125 | 182 | 8,308 | 8,075 | 360 | 8,435 |
| CORRECT | 21,025 | 1,707 | 22,732 | 19,480 | 1,133 | 20,614 |
| Step-by-Step | 109,979 | 1,092 | 111,071 | 31,188 | 767 | 31,955 |
| Who&When | LongRCA | |||||||||||
| Alg. | Hand. | Overall | ||||||||||
| Variant | Agent | Step | Agent | Step | Agent | Step | Tok. (K) | Role | Exact | 5 | MAE | Tok. (K) |
| DeepSeek-V4.1-Flash | ||||||||||||
| CoLFA | 65.87 | 52.38 | 62.07 | 27.59 | 64.67 | 44.57 | 6.9 | 51.58 | 21.84 | 32.81 | 35.54 | 22.4 |
| w/o Expansion | 61.11 | 46.83 | 56.90 | 22.41 | 59.78 | 39.13 | 7.6 | 47.02 | 19.65 | 30.53 | 37.91 | 32.9 |
| w/o Organization | 61.90 | 47.62 | 58.62 | 24.14 | 60.87 | 40.22 | 7.5 | 49.47 | 19.30 | 29.47 | 37.80 | 30.1 |
| Dataset | Backbone | CoLFA (%) | Best Baseline (%) | (%) | 95% CI (%) | |
|---|---|---|---|---|---|---|
| Who&When | DeepSeek-V4.1-Flash | 44.57 | CORRECT | (41.30) | ||
| Who&When | GLM-5.3-Flash | 46.74 | CORRECT | (44.57) | ||
| LongRCA | DeepSeek-V4.1-Flash | 21.84 | CORRECT | (18.86) | ||
| LongRCA | GLM-5.3-Flash | 23.60 | RAFFLES | (21.75) | ||
| Coverage | ||||
| Source | Other | Formation | Recent | Random |
| SWE-bench Pro | 35.2 | 68.2 | 15.9 | 35.5 |
| Terminal-Bench 2 | 52.5 | 52.4 | 23.8 | 38.0 |
| TravelPlanner | 67.3 | 71.7 | 55.9 | 49.4 |
| WebArena Verified | 81.9 | 78.6 | 67.6 | 56.2 |
| LongRCA (four sources combined) | 65.3 | 72.4 | 54.8 | 49.6 |
| Share limit | Coverage | Recent | Gap | ||
|---|---|---|---|---|---|
| 1.0 | 37 | 50.0 | 79.7 | 52.6 | 27.1 |
| 1.2 | 30 | 49.7 | 79.4 | 52.4 | 27.0 |
| 1.4 | 25 | 49.3 | 78.9 | 52.1 | 26.7 |
| 1.6 | 20 | 48.7 | 78.3 | 51.6 | 26.7 |
| 1.8 | 17 | 47.7 | 77.2 | 51.1 | 26.1 |
| Share (%) | |||||
|---|---|---|---|---|---|
| Source | Traces | Steps | Actions | Tool outputs or environment feedback | Total |
| TravelPlanner | 100 | 18.6 | 7.8 | 28.1 | 20.9 |
| WebArena Verified | 100 | 56.1 | 62.0 | 61.9 | 62.0 |
| SWE-bench Pro | 100 | 35.1 | 9.2 | 11.9 | 11.3 |
| Terminal-Bench 2 | 97.6 | 28.3 | 12.2 | 13.1 | 12.8 |
| All | 99.9 | 25.2 | 24.7 | 23.4 | 24.0 |
| Tokens (K) | Ratio (%) | |||
|---|---|---|---|---|
| Group | Trace | Input | Aggregate | Median |
| Shortest quintile | 5.8–30.8 | 12.1 | 58 | 61 |
| Second quintile | 30.8–38.2 | 12.5 | 44 | 36 |
| Middle quintile | 38.2–51.3 | 22.1 | 50 | 49 |
| Fourth quintile | 51.3–72.0 | 25.9 | 43 | 44 |
| Longest quintile | 72.1–394.2 | 34.2 | 31 | 32 |
| Coverage | DeepSeek | GLM | ||||||
|---|---|---|---|---|---|---|---|---|
| Strategy | All | Other agents | Exact | Tok. | CoP | Exact | Tok. | CoP |
| CoLFA | 78.9 | 72.4 | 21.12 | 23.3 | 110 | 22.97 | 22.9 | 100 |
| Recent | 61.5 | 44.8 | 17.93 | 21.2 | 118 | 18.12 | 21.0 | 116 |
| Random | 57.1 | 37.7 | 17.15 | 20.3 | 119 | 16.18 | 20.2 | 125 |