Real Faults in Deep Learning Fault Benchmarks: How Real Are They?
Authors: Gunel Jahangirova, Nargiz Humbatova, Jinhan Kim, Shin Yoo, Paolo Tonella
Organizations: Department of Informatics, King’s College London, London, UK · Software Institute, Università della Svizzera italiana (USI), Lugano, Switzerland · School of Computing, KAIST, Daejeon, Republic of Korea
As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 305 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 15.4% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 54% of cases.
Figures & tables
Dataset
Venue
Goal
#Bugs
TFBugs2018 [ 16 ]
ISSTA’18 (C)
Fault Analysis
151 (21)
DeepLocalize [ 24 ]
ICSE’21 (C)
Fault Localisation
41 (38)
DeepFD [ 4 ]
ICSE’22 (C)
Fault Localisation
52 (36)
Defects4ML [ 7 ]
EMSE’23 (J)
Fault Dataset
100 (68)
ISSRE’21 (C)
SFData [ 13 , 12 ]
IST’22 (J)
Fault Repair
146 (142)
Table 1: List of analysed benchmarks
Figure 1: An example of a case where the buggy version in the benchmark does not correspond to the buggy version provided in StackOverflow post 58609115.
Figure 2: An example of non-representative artificial training data from StackOverflow post 31880720 used in DeepFD benchmark.
Figure 3: Manual analysis of the benchmarks.
Dataset
#Bugs
Buggy m.
Fix m.
Buggy & fix m.
TD m.
TD real
Buggy & fix m. & Real TD m.
TFBugs2018
21
16 (76.2%)
19 (90.5%)
16 (76.2%)
11 (52.4%)
9 (42.9%)
4 (19.0%)
DeepLocalize
38
23 (60.5%)
21 (55.3%)
13 (34.2%)
24 (63.2%)
29 (76.3%)
11 (28.9%)
DeepFD
36
14 (38.9%)
14 (38.9%)
6 (16.7%)
29 (80.6%)
22 (61.1%)
2 (5.6%)
Defects4ML
68
58 (85.3%)
54 (79.4 %)
48 (70.6%)
53 (77.9%)
47 (69.1%)
29 (42.60%)
SFData
142
50 (35.2%)
114 (80.3%)
44 (31.0%)
102 (71.8%)
23 (16.2%)
1 (0.7%)
Overall
305
161 (52.8%)
222 (72.8%)
127 (41.6%)
219 (71.8%)
130 (42.6%)
47 (15.4%)
Table 2: Results on the correspondence of the faults in each benchmark
Dataset
Max
Average
Overall
#FT
TFBugs2018
3 (2)
1.2 (1.3)
25 (5)
14 (4)
DeepLocalize
6 (4)
2.5 (2.3)
93 (25)
14 (9)
DeepFD
8 (4)
3.2 (2.5)
114 (5)
15 (3)
Defects4ML
13 (7)
2.1 (2.1)
142 (60)
37 (26)
SFData
3 (3)
1.22 (1.3)
173 (56)
25 (17)
ALL
13 (7)
1.8 (1.7)
547 (151)
51 (39)
Table 3: Order of the bugs and fault types
Dataset
RSB A.
RSB R.
RSFC A.
RSFC R.
TFBugs2018
0/21 (0%)
0/4 (0%)
0/25 (0%)
0/5 (0%)
DeepLocalize
19/38 (50%)
6/11 (55%)
64/93 (69%)
18/25 (72%)
DeepFD
19/36 (53%)
2/2 (100%)
89/114 (78%)
5/5 (100%)
Defects4ML
23/68 (34%)
8/29 (28%)
61/142 (43%)
26/60 (43%)
SFData
1/142 (1%)
0/44 (0%)
4/173 (24%)
0/56 (0%)
ALL
62/305 (20%)
16/90 (18%)
218/547 (40%)
49/151 (32%)
Table 4: Results on similarity to artificial faults
Dataset
#Bugs
#Version Problems
#Reproduced
#Stable
TFBugs2018
4
4
0
N/A
DeepLocalize
11
2
6
3
DeepFD
2
0
2
2
Defects4ML
29
2
12
12
SFData
44
11
29
29
ALL
90
19
49
46
Table 5: Reproducibility Results
Dataset
Task Type
Framework
C
R
O
Keras
TF
TFBugs2018
15 (79%)
4 (21%)
0 (0%)
0 (0%)
21 (100%)
DeepLocalize
28 (74%)
9 (24%)
1 (2%)
38 (100%)
0 (0%)
DeepFD
26 (74%)
9 (26%)
0 (0%)
36 (100%)
0 (0%)
Defects4ML
53 (78%)
9 (13%)
6 (9%)
57 (84%)
11 (16%)
SFData
96 (73%)
14 (11%)
21 (16%)
85 (60%)
57 (40%)
Table 6: Results on performed task types and used frameworks