Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks Article Swipe

MAKE IMPACT

Qingrong Cheng , Keyu Wen , Xiaodong Gu · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2208.09596

Text-to-image synthesis aims to generate a photo-realistic and semantic consistent image from a specific text description. The images synthesized by off-the-shelf models usually contain limited components compared with the corresponding image and text description, which decreases the image quality and the textual-visual consistency. To address this issue, we propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*, which introduces a dual vision-language matching mechanism to strengthen the image quality and semantic consistency. The dual vision-language matching mechanism considers textual-visual matching between the generated image and the corresponding text description, and visual-visual consistent constraints between the synthesized image and the real image. Given a specific text description, VLMGAN* firstly encodes it into textual features and then feeds them to a dual vision-language matching-based generative model to synthesize a photo-realistic and textual semantic consistent image. Besides, the popular evaluation metrics for text-to-image synthesis are borrowed from simple image generation, which mainly evaluates the reality and diversity of the synthesized images. Therefore, we introduce a metric named Vision-Language Matching Score (VLMS) to evaluate the performance of text-to-image synthesis which can consider both the image quality and the semantic consistency between synthesized image and the description. The proposed dual multi-level vision-language matching strategy can be applied to other text-to-image synthesis methods. We implement this strategy on two popular baselines, which are marked with ${\text{VLMGAN}_{+\text{AttnGAN}}}$ and ${\text{VLMGAN}_{+\text{DFGAN}}}$. The experimental results on two widely-used datasets show that the model achieves significant improvements over other state-of-the-art methods.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2208.09596

PDF

https://arxiv.org/pdf/2208.09596

OA Status

green

Cited By

4

Related Works

10

OpenAlex ID

https://openalex.org/W4292957667

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4292957667

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2208.09596

Digital Object Identifier

Title

Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-08-20

Full publication date if available

Authors

Qingrong Cheng, Keyu Wen, Xiaodong Gu

List of authors in order

Landing page

https://arxiv.org/abs/2208.09596

Publisher landing page

PDF URL

https://arxiv.org/pdf/2208.09596

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2208.09596

Direct OA link when available

Concepts

Computer science, Consistency (knowledge bases), Image (mathematics), Artificial intelligence, Matching (statistics), Generative grammar, Image synthesis, Dual (grammatical number), Semantics (computer science), Metric (unit), Natural language processing, Computer vision, Information retrieval, Pattern recognition (psychology), Linguistics, Mathematics, Programming language, Statistics, Economics, Philosophy, Operations management

Top concepts (fields/topics) attached by OpenAlex

Cited by

4

Total citation count in OpenAlex

Citations by year (recent)

2025: 3, 2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload