ConvNets vs. Transformers: Whose Visual Representations are More Transferable? Article Swipe

MAKE IMPACT

Hong-Yu Zhou , Chixiang Lu , Sibei Yang , Yizhou Yu · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2108.05305

Vision transformers have attracted much attention from computer vision researchers as they are not restricted to the spatial inductive bias of ConvNets. However, although Transformer-based backbones have achieved much progress on ImageNet classification, it is still unclear whether the learned representations are as transferable as or even more transferable than ConvNets' features. To address this point, we systematically investigate the transfer learning ability of ConvNets and vision transformers in 15 single-task and multi-task performance evaluations. Given the strong correlation between the performance of pre-trained models and transfer learning, we include 2 residual ConvNets (i.e., R-101x3 and R-152x4) and 3 Transformer-based visual backbones (i.e., ViT-B, ViT-L and Swin-B), which have close error rates on ImageNet, that indicate similar transfer learning performance on downstream datasets. We observe consistent advantages of Transformer-based backbones on 13 downstream tasks (out of 15), including but not limited to fine-grained classification, scene recognition (classification, segmentation and depth estimation), open-domain classification, face recognition, etc. More specifically, we find that two ViT models heavily rely on whole network fine-tuning to achieve performance gains while Swin Transformer does not have such a requirement. Moreover, vision transformers behave more robustly in multi-task learning, i.e., bringing more improvements when managing mutually beneficial tasks and reducing performance losses when tackling irrelevant tasks. We hope our discoveries can facilitate the exploration and exploitation of vision transformers in the future.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2108.05305

PDF

https://arxiv.org/pdf/2108.05305

OA Status

green

Cited By

4

References

27

Related Works

10

OpenAlex ID

https://openalex.org/W3190732212

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3190732212

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2108.05305

Digital Object Identifier

Title

ConvNets vs. Transformers: Whose Visual Representations are More Transferable?

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-08-11

Full publication date if available

Authors

Hong-Yu Zhou, Chixiang Lu, Sibei Yang, Yizhou Yu

List of authors in order

Landing page

https://arxiv.org/abs/2108.05305

Publisher landing page

PDF URL

https://arxiv.org/pdf/2108.05305

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2108.05305

Direct OA link when available

Concepts

Transformer, Computer science, Artificial intelligence, Segmentation, Machine learning, Transferability, Transfer of learning, Multi-task learning, Task (project management), Voltage, Engineering, Logit, Systems engineering, Electrical engineering

Top concepts (fields/topics) attached by OpenAlex

Cited by

4

Total citation count in OpenAlex

Citations by year (recent)

2025: 2, 2024: 1, 2021: 1

Per-year citation counts (last 5 years)

References (count)

27

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload