Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones Article Swipe

MAKE IMPACT

Cheng Cui , Ruoyu Guo , Yuning Du , Dongliang He , Fu Li , Zewu Wu , Qiwen Liu , Shilei Wen , Jizhou Huang , Xiaoguang Hu , Dianhai Yu , Errui Ding , Yanjun Ma · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2103.05959

Recently, research efforts have been concentrated on revealing how pre-trained model makes a difference in neural network performance. Self-supervision and semi-supervised learning technologies have been extensively explored by the community and are proven to be of great potential in obtaining a powerful pre-trained model. However, these models require huge training costs (i.e., hundreds of millions of images or training iterations). In this paper, we propose to improve existing baseline networks via knowledge distillation from off-the-shelf pre-trained big powerful models. Different from existing knowledge distillation frameworks which require student model to be consistent with both soft-label generated by teacher model and hard-label annotated by humans, our solution performs distillation by only driving prediction of the student model consistent with that of the teacher model. Therefore, our distillation setting can get rid of manually labeled data and can be trained with extra unlabeled data to fully exploit capability of teacher model for better learning. We empirically find that such simple distillation settings perform extremely effective, for example, the top-1 accuracy on ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be significantly improved from 75.2% to 79% and 79.1% to 83%, respectively. We have also thoroughly analyzed what are dominant factors that affect the distillation performance and how they make a difference. Extensive downstream computer vision tasks, including transfer learning, object detection and semantic segmentation, can significantly benefit from the distilled pretrained models. All our experiments are implemented based on PaddlePaddle, codes and a series of improved pretrained models with ssld suffix are available in PaddleClas.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2103.05959

PDF

https://arxiv.org/pdf/2103.05959

OA Status

green

Cited By

2

References

45

Related Works

10

OpenAlex ID

https://openalex.org/W3135680385

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3135680385

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2103.05959

Digital Object Identifier

Title

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-03-10

Full publication date if available

Authors

Cheng Cui, Ruoyu Guo, Yuning Du, Dongliang He, Fu Li, Zewu Wu, Qiwen Liu, Shilei Wen, Jizhou Huang, Xiaoguang Hu, Dianhai Yu, Errui Ding, Yanjun Ma

List of authors in order

Landing page

https://arxiv.org/abs/2103.05959

Publisher landing page

PDF URL

https://arxiv.org/pdf/2103.05959

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2103.05959

Direct OA link when available

Concepts

Computer science, Distillation, Machine learning, Exploit, Artificial intelligence, Set (abstract data type), Artificial neural network, Simple (philosophy), Segmentation, Philosophy, Programming language, Chemistry, Computer security, Epistemology, Organic chemistry

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2025: 1, 2021: 1

Per-year citation counts (last 5 years)

References (count)

45

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload