Trouble With Time: Compile Timestamps and First-Seen Date Regression in EMBER2024
Organizations: CrowdStrike
Abstract
Reliable first-seen dates are important for constructing temporal malware evaluation splits, but this information often comes from external services that are not equally accessible to all researchers. We investigate whether machine learning can instead estimate an executable's first-seen date or determine when its PE/COFF compilation timestamp is unreliable. Using EMBER2024, we evaluate feature-based and raw-byte models across classification, regression, ranking, and latent-reliability formulations. Direct first-seen regression produces errors comparable to a substantial portion of the dataset's collection window, and none of the evaluated scores provides a useful chronological ordering of files. Current models therefore cannot replace externally sourced first-seen metadata. In contrast, classifiers can identify timestamps that fall within specified tolerances of first seen and can detect clearly implausible timestamps with high discrimination. We also find substantial timestamp disagreement among both benign and malicious executables, indicating that unreliable timestamps are not exclusively a product of malicious manipulation. These results establish baselines and identify timestamp reliability as a distinct direction for future security-ML research.
Figures & tables
| Threshold | Raw share | No-COFF AUC | No-COFF F1 | With-COFF AUC | With-COFF F1 |
|---|---|---|---|---|---|
| 30d | 0.2211 | 0.9705 | 0.8337 | 0.9756 | 0.8596 |
| 60d | 0.2459 | 0.9762 | 0.8606 | 0.9734 | 0.8863 |
| 90d | 0.2623 | 0.9774 | 0.8690 | 0.9743 | 0.9039 |
| 180d | 0.2967 | 0.9810 | 0.8947 | 0.9849 | 0.9419 |
| Threshold | Raw share | No-COFF AUC | No-COFF F1 | With-COFF AUC | With-COFF F1 |
|---|---|---|---|---|---|
| 30d | 0.3504 | 0.9734 | 0.8085 | 0.9794 | 0.9378 |
| 60d | 0.4132 | 0.9671 | 0.8270 | 0.9844 | 0.9448 |
| 90d | 0.4579 | 0.9680 | 0.8634 | 0.9595 | 0.9450 |
| 180d | 0.5587 | 0.9810 | 0.8967 | 0.9832 | 0.9796 |
| Target/loss | Representation | COFF as predictor | MAE | MedAE | p90 |
|---|---|---|---|---|---|
| raw timestamp | COFF baseline | – | 6,396.68 | 2,681.86 | 20,026.00 |
| ordinal abs-delta | XGBoost | no | 2,747.66 | 1,486.89 | 7,163.55 |
| ordinal abs-delta | XGBoost | yes | 2,044.67 | 1,147.65 | 7,593.38 |
| ordinal abs-delta | MalConv2 | raw bytes | 3,294.74 | 1,548.88 | 9,886.17 |
| signed delta | XGBoost | no | 2,029.49 | 274.33 | 5,351.69 |
| signed delta | XGBoost | yes | 296.33 | 154.05 | 781.46 |
| Target/loss | Representation | COFF as predictor | MAE | MedAE | p90 |
|---|---|---|---|---|---|
| raw timestamp | COFF baseline | – | 2,591.26 | 127.01 | 10,817.47 |
| ordinal abs-delta | XGBoost | no | 3,008.55 | 3,103.88 | 4,685.11 |
| ordinal abs-delta | XGBoost | yes | 1,453.19 | 1,119.91 | 2,169.12 |
| ordinal abs-delta | MalConv2 | raw bytes | 6,953.36 | 1,451.44 | 20,000.93 |
| signed delta | XGBoost | no | 1,997.83 | 1,056.79 | 5,150.41 |
| signed delta | XGBoost | yes | 311.82 | 181.83 | 716.57 |
| Split | Score family | Representation | Pair acc. | Spearman | Median rank err. | Median position-time err. |
|---|---|---|---|---|---|---|
| test | raw timestamp | COFF baseline | 0.5346 | 0.1041 | 0.2822 | 23.73 |
| test | latent age score | PyMC-BART | 0.5334 | 0.0852 | 0.2789 | 23.44 |
| test | within90 probability | XGBoost no-COFF | 0.5287 | 0.0838 | 0.2591 | 21.83 |
| test | within90 probability | XGBoost with-COFF | 0.5208 | 0.0603 | 0.2739 | 23.10 |
| test | ordinal abs-delta | XGBoost no-COFF | 0.5109 | 0.0309 | 0.2832 | 23.88 |
| test | ordinal abs-delta | XGBoost with-COFF | 0.5103 | 0.0298 | 0.2803 | 23.63 |
| Group | Mean | Median | Share | Median | |
|---|---|---|---|---|---|
| days | |||||
| all | 2,000 | 0.2641 | 0.0000 | 0.2640 | 2.39 |
| not obviously bad | 1,329 | 0.3975 | 0.0060 | 0.3973 | 2.47 |
| bad rule | 671 | 0.0000 | 0.0000 | 0.0000 | 2.20 |
| benign | 1,033 | 0.3154 | 0.0002 | 0.3175 | 2.78 |
| malicious | 967 | 0.2093 | 0.0000 | 0.2068 | 2.03 |
| Split | Label | Obvious-bad | Rate | |
|---|---|---|---|---|
| train | benign | 1,170,000 | 282,924 | 0.2418 |
| train | malicious | 1,170,000 | 531,602 | 0.4544 |
| test | benign | 270,000 | 88,033 | 0.3260 |
| test | malicious | 270,000 | 129,198 | 0.4785 |
| Split | Label | Zero | After first seen | Before 1995 | Repeated |
|---|---|---|---|---|---|
| train | benign | 0.2122 | 0.4865 | 0.3499 | 0.4277 |
| train | malicious | 0.1189 | 0.1419 | 0.3827 | 0.8604 |
| test | benign | 0.4226 | 0.4140 | 0.5131 | 0.5149 |
| test | malicious | 0.2246 | 0.1559 | 0.4693 | 0.8753 |
| Split | Group | Bad rate | AUROC | AUPRC | Brier | [email protected] | |
|---|---|---|---|---|---|---|---|
| test | all | 539,869 | 0.4024 | 0.9943 | 0.9921 | 0.0318 | 0.9341 |
| test | benign | 269,873 | 0.3262 | 0.9958 | 0.9929 | 0.0176 | 0.9672 |
| test | malicious | 269,996 | 0.4785 | 0.9923 | 0.9916 | 0.0459 | 0.9098 |
| challenge | malicious | 4,868 | 0.3640 | 0.9970 | 0.9946 | 0.0356 | 0.9348 |
| Feature family | Benign signed | Malicious signed | |
|---|---|---|---|
| header | -0.6775 | -0.0152 | 0.6623 |
| section | -0.4576 | 0.1136 | 0.5712 |
| imports | -0.3140 | 0.1834 | 0.4973 |
| Rich header | -0.3393 | -0.0810 | 0.2583 |
| strings | -0.1188 | 0.0476 | 0.1664 |
| authenticode | 0.0649 | -0.0986 | -0.1634 |
| Rank | Feature | Column | Benign | Malicious | |
|---|---|---|---|---|---|
| 1 | Authenticode signing-time minus COFF time | 2478 | 0.2658 | -0.1728 | -0.4386 |
| 2 | Debug directory size | 2416 | 0.3000 | -0.0955 | -0.3956 |
| 3 | Optional header uninitialized-data size | 714 | -0.0907 | 0.1665 | 0.2572 |
| 4 | Authenticode latest signing time | 2477 | -0.1541 | 0.0778 | 0.2319 |
| 5 | PE warning: suspicious section flags | 2546 | -0.0641 | 0.0710 | 0.1351 |
| 6 | Optional header major linker version | 704 | -0.2105 | -0.0776 | 0.1329 |
Malware Analysis
Malware analysis studies how to detect, classify, attribute, and understand malicious software and its behavior.
See all 66 papers