A common requirement across organizations is to have a tool that can identify file types based on their contents, particularly in the cybersecurity domain where magic numbers and file extensions can not be trusted. While existing tools work well in practice, there is plenty of room for improvement either in terms of computational load and time for detection in the case of model based tools like Magika or in terms of accuracy of detection in the case of file parsing tools that use programming language constructs. In this study, we propose a neural network model for identification of types of text content files, especially source code, that is more accurate and faster than other available tools. Our experiments on open-source files indicate that it is not only more accurate on average for text-content file-type identification, but also approximately four times faster than Magika, while being 28% smaller in size.
Figures & tables
Fig. 1: Magika model architecture from [ 1 ] . It works on three byte sequences extracted from the start, middle, and end of the files. The numbers next to the layers’ names indicate the size of their outputs.
Fig. 2: The dual-input convolutional neural network architecture of the Labeler model. The first three rows of blocks show how file contents are processed and the remaining blocks show the layers in the neural network model. The initial layers: embedding, spatial dropout, and 1-D Convolution, followed by layer normalization and dropout process, share the parameters, though they process the two token sequences separately.
Fig. 3: Inference time benchmarked by setting Labeler to use one CPU core only, without any such limitation on Magika. To measure Labeler inference times, a random byte array is generated of 8 Kb size and fed to a Rust implementation of the custom tokenizer the output of which is fed to the neural network model. Similarly, to measure Magika’s inference times, a random byte array of size 1.5 Kb is generated and passed as input to the Python interface of Magika. The average time taken for hundred thousand such calls is shown here
Metrics
Labeler
Magika
Macro F1
0.9804
0.9057
Weighted F1
0.982
0.921
Inference time per file
1.05 ms
3.85 ms
Model parameters size
715 Kb
1 Mb
TABLE I: Summary of performance metrics comparing Labeler and Magika. Labeler demonstrates superior accuracy with both macro and weighted F1-scores. Most significantly, Labeler has 3.7x faster inference speed even when it is limited to use a single CPU core, unlike Magika.
Language
Train Size
Test Size
F1-Score
Labeler
Magika
actionscript
103768
11497
0.9895
-
applescript
11214
1246
0.9730
-
asp
65067
7259
0.9881
0.4893
assembly
178619
19981
0.9896
0.9171
batchfile
188072
20967
0.9565
0.9207
TABLE II: Performance comparison for some of the supported file types. Bold values indicate the best results. – indicate unsupported types.
Fig. 4: Distribution of F1 score for the different file types. As seen, Labeler has a much narrower distribution than Magika. Both model gave least F1-scores for “text” file type. However, for other types, the results for Labeler show an F1-Score of at least 0.94.
Fig. 5: Improvement in precision and recall of Labeler with increase in training set size. Even with a very low number of files per type in the training set, Labeler quickly produces a higher accuracy than Magika using all available training data. This shows our method has excellent sample efficiency for rare and future new file types.
Fig. 6: Relation of precision and recall for the rarest five file types (DM, COBOL, R, VHDL, and Objective-C++) with increase in training set size. The drop in recall beyond a size of 50000 is likely due to the fact that all these types have less than 50000 files in the original training set.