A Close Look at Spatial Modeling: From Attention to Convolution Article Swipe

MAKE IMPACT

Xu Ma , Huan Wang , Can Qin , Kunpeng Li , Xingchen Zhao , Jie Fu , Yun Fu · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2212.12552

Vision Transformers have shown great promise recently for many vision tasks due to the insightful architecture design and attention mechanism. By revisiting the self-attention responses in Transformers, we empirically observe two interesting issues. First, Vision Transformers present a queryirrelevant behavior at deep layers, where the attention maps exhibit nearly consistent contexts in global scope, regardless of the query patch position (also head-irrelevant). Second, the attention maps are intrinsically sparse, few tokens dominate the attention weights; introducing the knowledge from ConvNets would largely smooth the attention and enhance the performance. Motivated by above observations, we generalize self-attention formulation to abstract a queryirrelevant global context directly and further integrate the global context into convolutions. The resulting model, a Fully Convolutional Vision Transformer (i.e., FCViT), purely consists of convolutional layers and firmly inherits the merits of both attention mechanism and convolutions, including dynamic property, weight sharing, and short- and long-range feature modeling, etc. Experimental results demonstrate the effectiveness of FCViT. With less than 14M parameters, our FCViT-S12 outperforms related work ResT-Lite by 3.7% top1 accuracy on ImageNet-1K. When scaling FCViT to larger models, we still perform better than previous state-of-the-art ConvNeXt with even fewer parameters. FCViT-based models also demonstrate promising transferability to downstream tasks, like object detection, instance segmentation, and semantic segmentation. Codes and models are made available at: https://github.com/ma-xu/FCViT.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2212.12552

PDF

https://arxiv.org/pdf/2212.12552

OA Status

green

Cited By

8

Related Works

10

OpenAlex ID

https://openalex.org/W4312225519

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4312225519

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2212.12552

Digital Object Identifier

Title

A Close Look at Spatial Modeling: From Attention to Convolution

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-12-23

Full publication date if available

Authors

Xu Ma, Huan Wang, Can Qin, Kunpeng Li, Xingchen Zhao, Jie Fu, Yun Fu

List of authors in order

Landing page

https://arxiv.org/abs/2212.12552

Publisher landing page

PDF URL

https://arxiv.org/pdf/2212.12552

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2212.12552

Direct OA link when available

Concepts

Computer science, Segmentation, Artificial intelligence, Convolutional neural network, Transformer, Convolution (computer science), Pattern recognition (psychology), Machine learning, Artificial neural network, Physics, Voltage, Quantum mechanics

Top concepts (fields/topics) attached by OpenAlex

Cited by

8

Total citation count in OpenAlex

Citations by year (recent)

2025: 1, 2024: 4, 2023: 3

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload