VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering Article Swipe

PDF

Chun-Mei Feng , Yang Bai , Tao Luo , Zhen Li , Salman Khan , Wangmeng Zuo , Xinxing Xu , Rick Siow Mong Goh , Yong Liu ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2312.12273

Albeit progress has been made in Composed Image Retrieval (CIR), we empirically find that a certain percentage of failure retrieval results are not consistent with their relative captions. To address this issue, this work provides a Visual Question Answering (VQA) perspective to boost the performance of CIR. The resulting VQA4CIR is a post-processing approach and can be directly plugged into existing CIR methods. Given the top-C retrieved images by a CIR method, VQA4CIR aims to decrease the adverse effect of the failure retrieval results being inconsistent with the relative caption. To find the retrieved images inconsistent with the relative caption, we resort to the "QA generation to VQA" self-verification pipeline. For QA generation, we suggest fine-tuning LLM (e.g., LLaMA) to generate several pairs of questions and answers from each relative caption. We then fine-tune LVLM (e.g., LLaVA) to obtain the VQA model. By feeding the retrieved image and question to the VQA model, one can find the images inconsistent with relative caption when the answer by VQA is inconsistent with the answer in the QA pair. Consequently, the CIR performance can be boosted by modifying the ranks of inconsistently retrieved images. Experimental results show that our proposed method outperforms state-of-the-art CIR methods on the CIRR and Fashion-IQ datasets.

Related Topics

Computer Science

Boosting (Machine Learning)

Perspective (Graphical)

Artificial Intelligence

Programming Language

Concepts

Question answering Computer science Boosting (machine learning) Pipeline (software) Information retrieval Perspective (graphical) Artificial intelligence Image retrieval Image (mathematics) Programming language

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2312.12273
PDF: https://arxiv.org/pdf/2312.12273
OA Status: green
Cited By: 2
Related Works: 10
OpenAlex ID: https://openalex.org/W4390050357

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4390050357

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2312.12273

Digital Object Identifier
Title: VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-12-19

Full publication date if available
Authors: Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman Khan, Wangmeng Zuo, Xinxing Xu, Rick Siow Mong Goh, Yong Liu

List of authors in order
Landing page: https://arxiv.org/abs/2312.12273

Publisher landing page
PDF URL: https://arxiv.org/pdf/2312.12273

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2312.12273

Direct OA link when available
Concepts: Question answering, Computer science, Boosting (machine learning), Pipeline (software), Information retrieval, Perspective (graphical), Artificial intelligence, Image retrieval, Image (mathematics), Programming language

Top concepts (fields/topics) attached by OpenAlex
Cited by: 2

Total citation count in OpenAlex
Citations by year (recent): 2024: 2

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4390050357
doi	https://doi.org/10.48550/arxiv.2312.12273
ids.doi	https://doi.org/10.48550/arxiv.2312.12273
ids.openalex	https://openalex.org/W4390050357
fwci
type	preprint
title	VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11714
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	1.0
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Multimodal Machine Learning Applications
topics[1].id	https://openalex.org/T10627
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9995999932289124
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1707
topics[1].subfield.display_name	Computer Vision and Pattern Recognition
topics[1].display_name	Advanced Image and Video Retrieval Techniques
topics[2].id	https://openalex.org/T11307
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9929999709129333
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Domain Adaptation and Few-Shot Learning
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C44291984
concepts[0].level	2
concepts[0].score	0.8390860557556152
concepts[0].wikidata	https://www.wikidata.org/wiki/Q1074173
concepts[0].display_name	Question answering
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.7591571807861328
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C46686674
concepts[2].level	2
concepts[2].score	0.711800754070282
concepts[2].wikidata	https://www.wikidata.org/wiki/Q466303
concepts[2].display_name	Boosting (machine learning)
concepts[3].id	https://openalex.org/C43521106
concepts[3].level	2
concepts[3].score	0.6566468477249146
concepts[3].wikidata	https://www.wikidata.org/wiki/Q2165493
concepts[3].display_name	Pipeline (software)
concepts[4].id	https://openalex.org/C23123220
concepts[4].level	1
concepts[4].score	0.49644309282302856
concepts[4].wikidata	https://www.wikidata.org/wiki/Q816826
concepts[4].display_name	Information retrieval
concepts[5].id	https://openalex.org/C12713177
concepts[5].level	2
concepts[5].score	0.4867737293243408
concepts[5].wikidata	https://www.wikidata.org/wiki/Q1900281
concepts[5].display_name	Perspective (graphical)
concepts[6].id	https://openalex.org/C154945302
concepts[6].level	1
concepts[6].score	0.47581249475479126
concepts[6].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[6].display_name	Artificial intelligence
concepts[7].id	https://openalex.org/C1667742
concepts[7].level	3
concepts[7].score	0.43929582834243774
concepts[7].wikidata	https://www.wikidata.org/wiki/Q10927554
concepts[7].display_name	Image retrieval
concepts[8].id	https://openalex.org/C115961682
concepts[8].level	2
concepts[8].score	0.4237244129180908
concepts[8].wikidata	https://www.wikidata.org/wiki/Q860623
concepts[8].display_name	Image (mathematics)
concepts[9].id	https://openalex.org/C199360897
concepts[9].level	1
concepts[9].score	0.0
concepts[9].wikidata	https://www.wikidata.org/wiki/Q9143
concepts[9].display_name	Programming language
keywords[0].id	https://openalex.org/keywords/question-answering
keywords[0].score	0.8390860557556152
keywords[0].display_name	Question answering
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.7591571807861328
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/boosting
keywords[2].score	0.711800754070282
keywords[2].display_name	Boosting (machine learning)
keywords[3].id	https://openalex.org/keywords/pipeline
keywords[3].score	0.6566468477249146
keywords[3].display_name	Pipeline (software)
keywords[4].id	https://openalex.org/keywords/information-retrieval
keywords[4].score	0.49644309282302856
keywords[4].display_name	Information retrieval
keywords[5].id	https://openalex.org/keywords/perspective
keywords[5].score	0.4867737293243408
keywords[5].display_name	Perspective (graphical)
keywords[6].id	https://openalex.org/keywords/artificial-intelligence
keywords[6].score	0.47581249475479126
keywords[6].display_name	Artificial intelligence
keywords[7].id	https://openalex.org/keywords/image-retrieval
keywords[7].score	0.43929582834243774
keywords[7].display_name	Image retrieval
keywords[8].id	https://openalex.org/keywords/image
keywords[8].score	0.4237244129180908
keywords[8].display_name	Image (mathematics)
language	en
locations[0].id	pmh:oai:arXiv.org:2312.12273
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2312.12273
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2312.12273
locations[1].id	doi:10.48550/arxiv.2312.12273
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2312.12273
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5049444898
authorships[0].author.orcid	https://orcid.org/0000-0002-3025-8964
authorships[0].author.display_name	Chun-Mei Feng
authorships[0].author_position	first
authorships[0].raw_author_name	Feng, Chun-Mei
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100762646
authorships[1].author.orcid	https://orcid.org/0000-0002-9092-3890
authorships[1].author.display_name	Yang Bai
authorships[1].author_position	middle
authorships[1].raw_author_name	Bai, Yang
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5101630043
authorships[2].author.orcid	https://orcid.org/0000-0002-3415-3676
authorships[2].author.display_name	Tao Luo
authorships[2].author_position	middle
authorships[2].raw_author_name	Luo, Tao
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5100332655
authorships[3].author.orcid	https://orcid.org/0000-0003-1166-9482
authorships[3].author.display_name	Zhen Li
authorships[3].author_position	middle
authorships[3].raw_author_name	Li, Zhen
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5058897625
authorships[4].author.orcid	https://orcid.org/0000-0002-9706-4005
authorships[4].author.display_name	Salman Khan
authorships[4].author_position	middle
authorships[4].raw_author_name	Khan, Salman
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5100636655
authorships[5].author.orcid	https://orcid.org/0000-0002-3330-783X
authorships[5].author.display_name	Wangmeng Zuo
authorships[5].author_position	middle
authorships[5].raw_author_name	Zuo, Wangmeng
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5067953198
authorships[6].author.orcid	https://orcid.org/0000-0003-1449-3072
authorships[6].author.display_name	Xinxing Xu
authorships[6].author_position	middle
authorships[6].raw_author_name	Xu, Xinxing
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5001599295
authorships[7].author.orcid	https://orcid.org/0000-0001-9116-1595
authorships[7].author.display_name	Rick Siow Mong Goh
authorships[7].author_position	middle
authorships[7].raw_author_name	Goh, Rick Siow Mong
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5100724297
authorships[8].author.orcid	https://orcid.org/0000-0003-4822-8939
authorships[8].author.display_name	Yong Liu
authorships[8].author_position	last
authorships[8].raw_author_name	Liu, Yong
authorships[8].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2312.12273
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11714
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	1.0
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Multimodal Machine Learning Applications
related_works	https://openalex.org/W2125652721, https://openalex.org/W1540371141, https://openalex.org/W4231274751, https://openalex.org/W2384605597, https://openalex.org/W1549363203, https://openalex.org/W2154063878, https://openalex.org/W2556012038, https://openalex.org/W1489772951, https://openalex.org/W1538046993, https://openalex.org/W4225922023
cited_by_count	2
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	2
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2312.12273
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2312.12273
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2312.12273
primary_location.id	pmh:oai:arXiv.org:2312.12273
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2312.12273
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2312.12273
publication_date	2023-12-19
publication_year	2023
referenced_works_count	0
abstract_inverted_index.a	14, 35, 51, 69
abstract_inverted_index.By	142
abstract_inverted_index.QA	111, 174
abstract_inverted_index.To	28, 90
abstract_inverted_index.We	131
abstract_inverted_index.be	56, 181
abstract_inverted_index.by	68, 165, 183
abstract_inverted_index.in	5, 172
abstract_inverted_index.is	50, 167
abstract_inverted_index.of	17, 45, 79, 123, 187
abstract_inverted_index.on	202
abstract_inverted_index.to	41, 74, 102, 106, 119, 137, 149
abstract_inverted_index.we	10, 100, 113
abstract_inverted_index."QA	104
abstract_inverted_index.CIR	61, 70, 178, 200
abstract_inverted_index.For	110
abstract_inverted_index.LLM	116
abstract_inverted_index.The	47
abstract_inverted_index.VQA	140, 151, 166
abstract_inverted_index.and	54, 125, 147, 205
abstract_inverted_index.are	21
abstract_inverted_index.can	55, 154, 180
abstract_inverted_index.has	2
abstract_inverted_index.not	22
abstract_inverted_index.one	153
abstract_inverted_index.our	195
abstract_inverted_index.the	43, 64, 76, 80, 87, 92, 97, 103, 139, 144, 150, 156, 163, 170, 173, 177, 185, 203
abstract_inverted_index.CIR.	46
abstract_inverted_index.CIRR	204
abstract_inverted_index.LVLM	134
abstract_inverted_index.VQA"	107
abstract_inverted_index.aims	73
abstract_inverted_index.been	3
abstract_inverted_index.each	128
abstract_inverted_index.find	12, 91, 155
abstract_inverted_index.from	127
abstract_inverted_index.into	59
abstract_inverted_index.made	4
abstract_inverted_index.show	193
abstract_inverted_index.that	13, 194
abstract_inverted_index.then	132
abstract_inverted_index.this	30, 32
abstract_inverted_index.when	162
abstract_inverted_index.with	24, 86, 96, 159, 169
abstract_inverted_index.work	33
abstract_inverted_index.(VQA)	39
abstract_inverted_index.Given	63
abstract_inverted_index.Image	7
abstract_inverted_index.being	84
abstract_inverted_index.boost	42
abstract_inverted_index.image	146
abstract_inverted_index.pair.	175
abstract_inverted_index.pairs	122
abstract_inverted_index.ranks	186
abstract_inverted_index.their	25
abstract_inverted_index.top-C	65
abstract_inverted_index.(CIR),	9
abstract_inverted_index.(e.g.,	117, 135
abstract_inverted_index.Albeit	0
abstract_inverted_index.LLaMA)	118
abstract_inverted_index.LLaVA)	136
abstract_inverted_index.Visual	36
abstract_inverted_index.answer	164, 171
abstract_inverted_index.effect	78
abstract_inverted_index.images	67, 94, 157
abstract_inverted_index.issue,	31
abstract_inverted_index.method	197
abstract_inverted_index.model,	152
abstract_inverted_index.model.	141
abstract_inverted_index.obtain	138
abstract_inverted_index.resort	101
abstract_inverted_index.VQA4CIR	49, 72
abstract_inverted_index.address	29
abstract_inverted_index.adverse	77
abstract_inverted_index.answers	126
abstract_inverted_index.boosted	182
abstract_inverted_index.caption	161
abstract_inverted_index.certain	15
abstract_inverted_index.failure	18, 81
abstract_inverted_index.feeding	143
abstract_inverted_index.images.	190
abstract_inverted_index.method,	71
abstract_inverted_index.methods	201
abstract_inverted_index.plugged	58
abstract_inverted_index.results	20, 83, 192
abstract_inverted_index.several	121
abstract_inverted_index.suggest	114
abstract_inverted_index.Composed	6
abstract_inverted_index.Question	37
abstract_inverted_index.approach	53
abstract_inverted_index.caption,	99
abstract_inverted_index.caption.	89, 130
abstract_inverted_index.decrease	75
abstract_inverted_index.directly	57
abstract_inverted_index.existing	60
abstract_inverted_index.generate	120
abstract_inverted_index.methods.	62
abstract_inverted_index.progress	1
abstract_inverted_index.proposed	196
abstract_inverted_index.provides	34
abstract_inverted_index.question	148
abstract_inverted_index.relative	26, 88, 98, 129, 160
abstract_inverted_index.Answering	38
abstract_inverted_index.Retrieval	8
abstract_inverted_index.captions.	27
abstract_inverted_index.datasets.	207
abstract_inverted_index.fine-tune	133
abstract_inverted_index.modifying	184
abstract_inverted_index.pipeline.	109
abstract_inverted_index.questions	124
abstract_inverted_index.resulting	48
abstract_inverted_index.retrieval	19, 82
abstract_inverted_index.retrieved	66, 93, 145, 189
abstract_inverted_index.Fashion-IQ	206
abstract_inverted_index.consistent	23
abstract_inverted_index.generation	105
abstract_inverted_index.percentage	16
abstract_inverted_index.empirically	11
abstract_inverted_index.fine-tuning	115
abstract_inverted_index.generation,	112
abstract_inverted_index.outperforms	198
abstract_inverted_index.performance	44, 179
abstract_inverted_index.perspective	40
abstract_inverted_index.Experimental	191
abstract_inverted_index.inconsistent	85, 95, 158, 168
abstract_inverted_index.Consequently,	176
abstract_inverted_index.inconsistently	188
abstract_inverted_index.post-processing	52
abstract_inverted_index.state-of-the-art	199
abstract_inverted_index.self-verification	108
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	9
citation_normalized_percentile