Linear Context Transform Block Article Swipe

MAKE IMPACT

Dongsheng Ruan , Jun Wen , Nenggan Zheng , Min Zheng · YOU? · · 2019 · Open Access · · DOI: https://doi.org/10.48550/arxiv.1909.03834

Squeeze-and-Excitation (SE) block presents a channel attention mechanism for modeling global context via explicitly capturing dependencies across channels. However, we are still far from understanding how the SE block works. In this work, we first revisit the SE block, and then present a detailed empirical study of the relationship between global context and attention distribution, based on which we propose a simple yet effective module, called Linear Context Transform (LCT) block. We divide all channels into different groups and normalize the globally aggregated context features within each channel group, reducing the disturbance from irrelevant channels. Through linear transform of the normalized context features, we model global context for each channel independently. The LCT block is extremely lightweight and easy to be plugged into different backbone models while with negligible parameters and computational burden increase. Extensive experiments show that the LCT block outperforms the SE block in image classification task on the ImageNet and object detection/segmentation on the COCO dataset with different backbone models. Moreover, LCT yields consistent performance gains over existing state-of-the-art detection architectures, e.g., 1.5$\sim$1.7% AP$^{bbox}$ and 1.0$\sim$1.2% AP$^{mask}$ improvements on the COCO benchmark, irrespective of different baseline models of varied capacities. We hope our simple yet effective approach will shed some light on future research of attention-based models.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/1909.03834

PDF

https://arxiv.org/pdf/1909.03834

OA Status

green

Cited By

2

References

40

Related Works

10

OpenAlex ID

https://openalex.org/W2972158181

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W2972158181

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.1909.03834

Digital Object Identifier

Title

Linear Context Transform Block

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2019

Year of publication

Publication date

2019-09-06

Full publication date if available

Authors

Dongsheng Ruan, Jun Wen, Nenggan Zheng, Min Zheng

List of authors in order

Landing page

https://arxiv.org/abs/1909.03834

Publisher landing page

PDF URL

https://arxiv.org/pdf/1909.03834

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/1909.03834

Direct OA link when available

Concepts

Block (permutation group theory), Context (archaeology), Benchmark (surveying), Computer science, Channel (broadcasting), Segmentation, Algorithm, Object detection, Artificial intelligence, Pattern recognition (psychology), Mathematics, Telecommunications, Cartography, Geometry, Geography, Archaeology

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2020: 2

Per-year citation counts (last 5 years)

References (count)

40

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload