I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification Article Swipe

MAKE IMPACT

Muhammad Ferjad Naeem , Yongqin Xian , Luc Van Gool , Federico Tombari · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2209.10304

Despite the tremendous progress in zero-shot learning(ZSL), the majority of existing methods still rely on human-annotated attributes, which are difficult to annotate and scale. An unsupervised alternative is to represent each class using the word embedding associated with its semantic class name. However, word embeddings extracted from pre-trained language models do not necessarily capture visual similarities, resulting in poor zero-shot performance. In this work, we argue that online textual documents, e.g., Wikipedia, contain rich visual descriptions about object classes, therefore can be used as powerful unsupervised side information for ZSL. To this end, we propose I2DFormer, a novel transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. In order to distill discriminative visual words from noisy documents, we introduce a new cross-modal attention module that learns fine-grained interactions between image patches and document words. Consequently, our I2DFormer not only learns highly discriminative document embeddings that capture visual similarities but also gains the ability to localize visually relevant words in image regions. Quantitatively, we demonstrate that our I2DFormer significantly outperforms previous unsupervised semantic embeddings under both zero-shot and generalized zero-shot learning settings on three public datasets. Qualitatively, we show that our method leads to highly interpretable results where document words can be grounded in the image regions.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2209.10304

PDF

https://arxiv.org/pdf/2209.10304

OA Status

green

Cited By

21

Related Works

10

OpenAlex ID

https://openalex.org/W4297899821

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4297899821

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2209.10304

Digital Object Identifier

Title

I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-09-21

Full publication date if available

Authors

Muhammad Ferjad Naeem, Yongqin Xian, Luc Van Gool, Federico Tombari

List of authors in order

Landing page

https://arxiv.org/abs/2209.10304

Publisher landing page

PDF URL

https://arxiv.org/pdf/2209.10304

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2209.10304

Direct OA link when available

Concepts

Computer science, Discriminative model, Embedding, Artificial intelligence, Image (mathematics), Class (philosophy), Pattern recognition (psychology), Natural language processing, Word (group theory), Contextual image classification, Machine learning, Mathematics, Geometry

Top concepts (fields/topics) attached by OpenAlex

Cited by

21

Total citation count in OpenAlex

Citations by year (recent)

2025: 6, 2024: 11, 2023: 4

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload