Image Fusion Transformer Article Swipe

MAKE IMPACT

Vibashan VS , Jeya Maria Jose Valanarasu , Poojan Oza , Vishal M. Patel · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2107.09011

In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information. In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion. Specifically, CNN-based methods perform image fusion by fusing local features. However, they do not consider long-range dependencies that are present in the image. Transformer-based models are designed to overcome this by modeling the long-range dependencies with the help of self-attention mechanism. This motivates us to propose a novel Image Fusion Transformer (IFT) where we develop a transformer-based multi-scale fusion strategy that attends to both local and long-range information (or global context). The proposed method follows a two-stage training approach. In the first stage, we train an auto-encoder to extract deep features at multiple scales. In the second stage, multi-scale features are fused using a Spatio-Transformer (ST) fusion strategy. The ST fusion blocks are comprised of a CNN and a transformer branch which capture local and long-range features, respectively. Extensive experiments on multiple benchmark datasets show that the proposed method performs better than many competitive fusion algorithms. Furthermore, we show the effectiveness of the proposed ST fusion strategy with an ablation analysis. The source code is available at: https://github.com/Vibashan/Image-Fusion-Transformer.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2107.09011

PDF

https://arxiv.org/pdf/2107.09011

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4300818588

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4300818588

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2107.09011

Digital Object Identifier

Title

Image Fusion Transformer

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-07-19

Full publication date if available

Authors

Vibashan VS, Jeya Maria Jose Valanarasu, Poojan Oza, Vishal M. Patel

List of authors in order

Landing page

https://arxiv.org/abs/2107.09011

Publisher landing page

PDF URL

https://arxiv.org/pdf/2107.09011

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2107.09011

Direct OA link when available

Concepts

Computer science, Artificial intelligence, Transformer, Image fusion, Encoder, Fusion, Pattern recognition (psychology), Convolutional neural network, Computer vision, Image (mathematics), Engineering, Voltage, Philosophy, Linguistics, Electrical engineering, Operating system

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2024: 1, 2021: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload