Abstract
In this paper, we propose a new heuristic training procedure to help a deep neural network (DNN) repeatedly escape from a local minimum and move to a better local minimum. Our method repeats the following processes multiple times: Randomly reinitializing the weights of the last layer of a converged DNN while preserving the weights of the remaining layers, and then conducting a new round of training. The motivation is to make the training in the new round learn better parameters based on the 'good' initial parameters learned in the previous round. With multiple randomly initialized DNNs trained based on our training procedure, we can obtain an ensemble of DNNs that are more accurate and diverse compared with the normal training procedure. We call this framework 'retraining'. Experiments on eight DNN models show that our method generally outperforms the state-of-the-art ensemble learning methods. We also provide two variants of the retraining framework to tackle the tasks of ensemble learning in which 1) DNNs exhibit very high training accuracies (e.g., > 95%) and 2) DNNs are too computationally expensive to train.
| Original language | English |
|---|---|
| Title of host publication | 2018 24th International Conference on Pattern Recognition, ICPR 2018 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 860-867 |
| Number of pages | 8 |
| ISBN (Electronic) | 9781538637883 |
| DOIs | |
| Publication status | Published - Nov 26 2018 |
| Event | 24th International Conference on Pattern Recognition, ICPR 2018 - Beijing, China Duration: Aug 20 2018 → Aug 24 2018 |
Publication series
| Name | Proceedings - International Conference on Pattern Recognition |
|---|---|
| Volume | 2018-August |
| ISSN (Print) | 1051-4651 |
Other
| Other | 24th International Conference on Pattern Recognition, ICPR 2018 |
|---|---|
| Country | China |
| City | Beijing |
| Period | 8/20/18 → 8/24/18 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Computer Vision and Pattern Recognition
Cite this
Retraining : A Simple Way to Improve the Ensemble Accuracy of Deep Neural Networks for Image Classification. / Zhao, Kaikai; Matsukawa, Tetsu; Suzuki, Einoshin.
2018 24th International Conference on Pattern Recognition, ICPR 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 860-867 8545535 (Proceedings - International Conference on Pattern Recognition; Vol. 2018-August).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Retraining
T2 - A Simple Way to Improve the Ensemble Accuracy of Deep Neural Networks for Image Classification
AU - Zhao, Kaikai
AU - Matsukawa, Tetsu
AU - Suzuki, Einoshin
PY - 2018/11/26
Y1 - 2018/11/26
N2 - In this paper, we propose a new heuristic training procedure to help a deep neural network (DNN) repeatedly escape from a local minimum and move to a better local minimum. Our method repeats the following processes multiple times: Randomly reinitializing the weights of the last layer of a converged DNN while preserving the weights of the remaining layers, and then conducting a new round of training. The motivation is to make the training in the new round learn better parameters based on the 'good' initial parameters learned in the previous round. With multiple randomly initialized DNNs trained based on our training procedure, we can obtain an ensemble of DNNs that are more accurate and diverse compared with the normal training procedure. We call this framework 'retraining'. Experiments on eight DNN models show that our method generally outperforms the state-of-the-art ensemble learning methods. We also provide two variants of the retraining framework to tackle the tasks of ensemble learning in which 1) DNNs exhibit very high training accuracies (e.g., > 95%) and 2) DNNs are too computationally expensive to train.
AB - In this paper, we propose a new heuristic training procedure to help a deep neural network (DNN) repeatedly escape from a local minimum and move to a better local minimum. Our method repeats the following processes multiple times: Randomly reinitializing the weights of the last layer of a converged DNN while preserving the weights of the remaining layers, and then conducting a new round of training. The motivation is to make the training in the new round learn better parameters based on the 'good' initial parameters learned in the previous round. With multiple randomly initialized DNNs trained based on our training procedure, we can obtain an ensemble of DNNs that are more accurate and diverse compared with the normal training procedure. We call this framework 'retraining'. Experiments on eight DNN models show that our method generally outperforms the state-of-the-art ensemble learning methods. We also provide two variants of the retraining framework to tackle the tasks of ensemble learning in which 1) DNNs exhibit very high training accuracies (e.g., > 95%) and 2) DNNs are too computationally expensive to train.
UR - http://www.scopus.com/inward/record.url?scp=85059759484&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85059759484&partnerID=8YFLogxK
U2 - 10.1109/ICPR.2018.8545535
DO - 10.1109/ICPR.2018.8545535
M3 - Conference contribution
AN - SCOPUS:85059759484
T3 - Proceedings - International Conference on Pattern Recognition
SP - 860
EP - 867
BT - 2018 24th International Conference on Pattern Recognition, ICPR 2018
PB - Institute of Electrical and Electronics Engineers Inc.
ER -