Learning From How Humans Correct
Abstract
In industry NLP application, our manually labeled data has a certain number of noisy data. We present a simple method to find the noisy data and relabel them manually, meanwhile we collect the correction information. Then we present novel method to incorporate the human correction information into deep learning model. Human know how to correct noisy data. So the correction information can be inject into deep learning model. We do the experiment on our own text classification dataset, which is manually labeled, because we need to relabel the noisy data in our dataset for our industry application. The experiment result shows that our learn-on-correction method improve the classification accuracy from 91.7% to 92.5% in test dataset. The 91.7% accuracy is trained on the corrected dataset, which improve the baseline from 83.3% to 91.7% in test dataset. The accuracy under human evaluation achieves more than 97%.
Figures & tables
| Data Size | Description |
|---|---|
| 500,000,000 | All the data in our application database. |
| 2,040,000 | The data we label in step 1 of Fig 1. |
| 2,000,000 | The the training dataset we split from the 2,040,000 data. |
| 300,000 | All the noisy data we select from the 2,040,000 data to relabel. |
| 294,120 | The noisy data in the training dataset. |
| 40,000 | The test dataset we split from the 2,040,000 data. |
| Model In Fig 1 | Test Dataset Accuracy | Human Evaluate Accuracy |
|---|---|---|
| Model-A | 83.3% | 88.0% |
| Model-B | 91.7% | 97.2% |
| Model-C | 92.5% | 97.7% |
Learning with Noisy Labels
Learning with noisy labels studies how to train reliable models when annotations are incorrect, incomplete, subjective, or weakly supervised.
See all 150 papers