Head-Free Lightweight Semantic Segmentation with Linear Transformer Article Swipe

MAKE IMPACT

Bo Dong , Pichao Wang , Fan Wang · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2301.04648

Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for semantic segmentation, named Adaptive Frequency Transformer. It adopts a parallel architecture to leverage prototype representations as specific learnable local descriptions which replaces the decoder and preserves the rich image semantics on high-resolution features. Although removing the decoder compresses most of the computation, the accuracy of the parallel structure is still limited by low computational resources. Therefore, we employ heterogeneous operators (CNN and Vision Transformer) for pixel embedding and prototype representations to further save computational costs. Moreover, it is very difficult to linearize the complexity of the vision Transformer from the perspective of spatial domain. Due to the fact that semantic segmentation is very sensitive to frequency information, we construct a lightweight prototype learning block with adaptive frequency filter of complexity $O(n)$ to replace standard self attention with $O(n^{2})$. Extensive experiments on widely adopted datasets demonstrate that our model achieves superior accuracy while retaining only 3M parameters. On the ADE20K dataset, our model achieves 41.8 mIoU and 4.6 GFLOPs, which is 4.4 mIoU higher than Segformer, with 45% less GFLOPs. On the Cityscapes dataset, our model achieves 78.7 mIoU and 34.4 GFLOPs, which is 2.5 mIoU higher than Segformer with 72.5% less GFLOPs. Code is available at https://github.com/dongbo811/AFFormer.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2301.04648

PDF

https://arxiv.org/pdf/2301.04648

OA Status

green

Cited By

9

Related Works

10

OpenAlex ID

https://openalex.org/W4315884020

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4315884020

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2301.04648

Digital Object Identifier

Title

Head-Free Lightweight Semantic Segmentation with Linear Transformer

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-01-11

Full publication date if available

Authors

Bo Dong, Pichao Wang, Fan Wang

List of authors in order

Landing page

https://arxiv.org/abs/2301.04648

Publisher landing page

PDF URL

https://arxiv.org/pdf/2301.04648

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2301.04648

Direct OA link when available

Concepts

Computer science, FLOPS, Segmentation, Transformer, Artificial intelligence, Computer vision, Computer engineering, Parallel computing, Voltage, Quantum mechanics, Physics

Top concepts (fields/topics) attached by OpenAlex

Cited by

9

Total citation count in OpenAlex

Citations by year (recent)

2025: 3, 2024: 6

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload