K-LITE: Learning Transferable Visual Models with External Knowledge Article Swipe

MAKE IMPACT

Sheng Shen , Chunyuan Li , Xiaowei Hu , Yujia Xie , Jianwei Yang , Pengchuan Zhang , Anna Rohrbach , Zhe Gan , Lijuan Wang , Lu Yuan , Ce Liu , Kurt Keutzer , Trevor Darrell , Jianfeng Gao · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2204.09222

The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at https://github.com/microsoft/klite.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2204.09222

PDF

https://arxiv.org/pdf/2204.09222

OA Status

green

Cited By

44

Related Works

10

OpenAlex ID

https://openalex.org/W4224232320

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4224232320

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2204.09222

Digital Object Identifier

Title

K-LITE: Learning Transferable Visual Models with External Knowledge

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-04-20

Full publication date if available

Authors

Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, Jianfeng Gao

List of authors in order

Landing page

https://arxiv.org/abs/2204.09222

Publisher landing page

PDF URL

https://arxiv.org/pdf/2204.09222

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2204.09222

Direct OA link when available

Concepts

Computer science, Leverage (statistics), Artificial intelligence, Generality, Usability, Benchmarking, Scalability, Human–computer interaction, Machine learning, Natural language processing, Database, Business, Marketing, Psychology, Psychotherapist

Top concepts (fields/topics) attached by OpenAlex

Cited by

44

Total citation count in OpenAlex

Citations by year (recent)

2025: 8, 2024: 13, 2023: 22, 2022: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload