An Extensive Evaluation of Machine Learning and Deep Learning Approaches for Sentiment Analysis: Insights from Amazon Customer Reviews
Organizations: Computer Science Dept., CUNY College of Staten Island · Computer Science Dept., CUNY LaGuardia Community College
Abstract
The rapid growth of user-generated text on online platforms creates opportunities for understanding customer opinion and supporting data-driven decisions. This study compares five traditional machine learning classifiers (Random Forest, Logistic Regression, Bernoulli Naive Bayes, Linear Support Vector Classifier, and Multinomial Naive Bayes) and six deep learning architectures (LSTM, Bi-LSTM, GRU, RNN, Attention, and FastText) for three-class sentiment classification of 568,454 Amazon Fine Food Reviews. Machine learning models used Bag-of-Words (BoW) and TF-IDF features limited to 5,000 terms; deep learning models used learned embeddings over a 20,000-word vocabulary. Models were evaluated on the original imbalanced data and on a random-oversampled balanced set of 1,070,469 reviews, using accuracy, macro and per-class precision, recall, and F1-score, and training time. On the balanced set, Random Forest with BoW achieved 99% accuracy and the Attention model achieved 98.7%. Linear SVC reached 89% accuracy in 1 min 15 s, and Logistic Regression reached 90% in 3 min 37 s. On the imbalanced set, the Attention model achieved 97.8% accuracy, and FastText achieved 95.3% in 3 min 26 s, while the RNN reached 91.9% with a neutral-class F1-score of 0.08. Deep learning models were more stable across class distributions, whereas the best classical model depended on balancing. We also report a user-facing application that applies the trained Random Forest and Attention models to new review text. Because the labels are derived from star ratings, the results measure agreement with rating-derived sentiment classes rather than annotated sentiment; the dataset is a single source, and the findings should be read with that limitation in mind.
Figures & tables
| Positive | Negative | Neutral | |||
| Word | Count | Word | Count | Word | Count |
| good | 200,638 | never | 39,275 | fine | 16,110 |
| great | 167,175 | bad | 32,186 | grocery store | 12,679 |
| love | 127,520 | disappointed | 16,624 | first time | 7,795 |
| even | 75,000 | taste like | 11,651 | normal | 7,265 |
| well | 73,408 | awful | 5,109 | even though | 6,755 |
| Positive | Negative | Neutral | |||
|---|---|---|---|---|---|
| Word | Weight | Word | Weight | Word | Weight |
| highly recommended | 0.8534 | never buy | -0.93 | not bad | 0.431 |
| love | 0.6369 | bad taste | -0.5423 | okay | 0.2263 |
| great | 0.6249 | bad | -0.5423 | fine | 0.2023 |
| great product | 0.6249 | disgusting | -0.5267 | average | 0.0 |
| delicious | 0.5719 | never | -0.5234 | normal | 0.0 |
| Step | ML pipeline | DL pipeline |
|---|---|---|
| Lowercasing | Yes | Yes |
| Removal of URLs, mentions, hashtags, digits, punctuation | Yes | HTML tags and non-alphabetic characters removed |
| Stopword removal | Yes (NLTK English list) | No |
| Stemming / lemmatization | No | No |
| Vectorization | BoW or TF-IDF, 5000 features | Integer sequences, padded to 100 |
| Parameter | RFC | LRC | L-SVC | BNB | MNB |
|---|---|---|---|---|---|
| Implementation | RandomForestClassifier | LogisticRegression | LinearSVC | BernoulliNB | MultinomialNB |
| Number of trees | n_estimators =5 | – | – | – | – |
| Max iterations | – | max_iter =100 | max_iter =1000 | – | – |
| Regularization | – | L2, =1.0 | L2, =1.0 | =1.0 | =1.0 |
| Loss / solver | Gini impurity | lbfgs | squared hinge | – | – |
| Class prior | – | – | – | training-split proportions | training-split proportions |
| Model | BoW | TF-IDF |
|---|---|---|
| Random Forest Classifier (RFC) | 0.99 | 0.98 |
| Logistic Regression Classifier (LRC) | 0.90 | 0.85 |
| Linear Support Vector Classifier (L-SVC) | 0.89 | 0.84 |
| Bernoulli Naive Bayes (BNB) | 0.76 | 0.74 |
| Multinomial Naive Bayes (MNB) | 0.79 | 0.76 |
| Model | BoW (Time) | TF-IDF (Time) |
|---|---|---|
| Random Forest Classifier (RFC) | 16m 55s | 17m 16s |
| Logistic Regression Classifier (LRC) | 3m 37s | 3m 36s |
| Linear Support Vector Classifier (L-SVC) | 1m 15s | 1m 14s |
| Bernoulli Naive Bayes (BNB) | 0m 29s | 0m 29s |
| Multinomial Naive Bayes (MNB) | 0m 6s | 0m 6s |
| Category | Model | Feature Extraction | Performance Metric |
| Fastest Model | Multinomial Naive Bayes | BoW | 0m 6s |
| Most Accurate Model | Random Forest Classifier | BoW | 0.99 Accuracy |
| Most Efficient Model | Linear Support Vector Classifier | BoW | E = T A |
| Classification Reports of All Models Across Dataset Sizes | |||||||||||
| Dataset Size | Model | Precision | Recall | F1 Score | Accuracy | ||||||
| Negative | Neutral | Positive | Negative | Neutral | Positive | Negative | Neutral | Positive | |||
| Imbalanced | RFC | 0.57 | 0.50 | 0.93 | 0.27 | 0.41 | 0.98 | 0.36 | 0.45 | 0.96 | 0.91 |
| LRC | 0.68 | 0.59 | 0.94 | 0.40 | 0.31 | 0.98 | 0.50 | 0.40 | 0.96 | 0.93 | |
| BNB | 0.41 | 0.13 | 0.96 | 0.44 | 0.81 | 0.87 | 0.43 | 0.22 | 0.91 | 0.84 | |
| L-SVC | 0.71 | 0.57 | 0.94 | 0.35 | 0.20 | 0.98 | 0.47 | 0.29 | 0.96 | 0.92 | |
| Classification Reports of All Models Across Dataset Sizes | |||||||||||
| Dataset Size | Model | Precision | Recall | F1 Score | Accuracy | ||||||
| Negative | Neutral | Positive | Negative | Neutral | Positive | Negative | Neutral | Positive | |||
| Imbalanced | RFC | 0.53 | 0.47 | 0.89 | 0.22 | 0.37 | 0.93 | 0.31 | 0.41 | 0.91 | 0.86 |
| LRC | 0.63 | 0.54 | 0.90 | 0.35 | 0.26 | 0.94 | 0.45 | 0.35 | 0.91 | 0.88 | |
| BNB | 0.38 | 0.10 | 0.91 | 0.39 | 0.76 | 0.83 | 0.38 | 0.18 | 0.86 | 0.79 | |
| L-SVC | 0.45 | 0.33 | 0.92 | 0.54 | 0.78 | 0.90 | 0.49 | 0.54 | 0.88 | 0.86 | |
| Model | Sequence layer | Pooling / attention | Dense head |
|---|---|---|---|
| LSTM | LSTM, 128 units | – | Dense 64, ReLU |
| Bi-LSTM | Bidirectional LSTM, 128 units | – | Dense 64, ReLU |
| GRU | GRU, 128 units | – | Dense 64, ReLU |
| RNN | SimpleRNN, 128 units | – | Dense 64, ReLU |
| Attention | LSTM, 128 units, return_sequences =True | Self-attention ( Attention layer on ), GlobalAveragePooling1D | Dense 64, ReLU |
| FastText-style | – | GlobalAveragePooling1D | Dense 64, ReLU |
| Model | Running Time | Accuracy |
|---|---|---|
| LSTM | 53m 51s | 0.984 |
| RNN | 34m 47s | 0.664 |
| Bi-LSTM | 95m 20s | 0.986 |
| GRU | 61m 37s | 0.981 |
| Attention | 112m 3s | 0.987 |
| FastText | 29m 54s | 0.979 |
| Category | Model | Performance Metric |
|---|---|---|
| Fastest Model | FastText | 29m 54s |
| Most Accurate Model | Attention | 0.987 Accuracy |
| Most Efficient Model | Bi-LSTM | E = T A |
| Panel A: Deep Learning Models Classification Report | ||||||||||
| Model | Precision | Recall | F1 Score | Accuracy | ||||||
| Negative | Neutral | Positive | Negative | Neutral | Positive | Negative | Neutral | Positive | ||
| Attention | 0.9048 | 0.9286 | 0.9849 | 0.8416 | 0.8661 | 0.9923 | 0.8721 | 0.8963 | 0.9886 | 0.9780 |
| 11m 6s | ||||||||||
| Bi-LSTM | 0.9022 | 0.9426 | 0.9837 | 0.8340 | 0.8478 | 0.9921 | 0.8668 | 0.8927 | 0.9879 | 0.9770 |
| 15m 27s | ||||||||||
| Panel A: Best ML (RFC) and Best DL (Attention) Models Classification Report | |||||||||||
| Dataset Size | Model | Precision | Recall | F1 Score | Accuracy | ||||||
| Negative | Neutral | Positive | Negative | Neutral | Positive | Negative | Neutral | Positive | |||
| Imbalanced | RFC | 0.5671 | 0.5005 | 0.9336 | 0.2689 | 0.4082 | 0.9809 | 0.3648 | 0.4497 | 0.9567 | 0.9143 |
| P/356,823 Neu/32,312 N/6,012 T/395,147 | Attention | 0.8805 | 0.9444 | 0.9873 | 0.8745 | 0.8336 | 0.9898 | 0.8775 | 0.8856 | 0.9885 | 0.9780 |
| 25% Dataset | RFC | 0.8400 | 0.9700 | 0.8700 | 0.8600 | 1.0000 | 0.8200 | 0.8500 | 0.9800 | 0.8400 | 0.8900 |
| P/Neu/N/32,928 T/98,784 | Attention | 0.8209 | 0.7787 | 0.8568 | 0.8270 | 0.7933 | 0.8344 | 0.8239 | 0.7859 | 0.8455 | 0.8182 |
| Category | Model | Performance Metric |
| Fastest Model | MNB (BoW) | 0m 6s |
| Most Accurate Model | RFC (BoW) | 0.99 Accuracy |
| Most Efficient Model | FastText | |
| Best Model | Attention | Accuracy Time |
Sentiment Analysis
Sentiment analysis detects and measures opinions, emotions, and evaluative stance in text and multimodal data.
See all 136 papers