Gou, J. and Yu, B. and Maybank, Stephen and Tao, Da. (2021) Knowledge distillation: a survey. International Journal of Computer Vision 129 , pp. 1789-1819. ISSN 0920-5691.
|
Text
KD_Survey-arxiv.pdf - Author's Accepted Manuscript Download (881kB) | Preview |
Abstract
In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparision and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.
Metadata
| Item Type: | Article |
|---|---|
| Keyword(s) / Subject(s): | Deep neural networks, Model compression, Knowledge distillation, Knowledge transfer, Teacher-student architecture |
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Depositing User: | Steve Maybank |
| Date Deposited: | 17 May 2021 14:11 |
| Last Modified: | 09 Aug 2023 12:50 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/44038 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
