Improving Selective Visual Question Answering by Learning from Your Peers Article Swipe

MAKE IMPACT

Corentin Dancette , Spencer Whitehead , Rishabh Maheshwary , Ramakrishna Vedantam , Stefan Scherer , Xinlei Chen , Matthieu Cord , Marcus Rohrbach · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2306.08751

Despite advances in Visual Question Answering (VQA), the ability of models to assess their own correctness remains underexplored. Recent work has shown that VQA models, out-of-the-box, can have difficulties abstaining from answering when they are wrong. The option to abstain, also called Selective Prediction, is highly relevant when deploying systems to users who must trust the system's output (e.g., VQA assistants for users with visual impairments). For such scenarios, abstention can be especially important as users may provide out-of-distribution (OOD) or adversarial inputs that make incorrect answers more likely. In this work, we explore Selective VQA in both in-distribution (ID) and OOD scenarios, where models are presented with mixtures of ID and OOD data. The goal is to maximize the number of questions answered while minimizing the risk of error on those questions. We propose a simple yet effective Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model. It does not require additional manual labels or held-out data and provides a signal for identifying examples that are easy/difficult to generalize to. In our extensive evaluations, we show this benefits a number of models across different architectures and scales. Overall, for ID, we reach 32.92% in the selective prediction metric coverage at 1% risk of error (C@1%) which doubles the previous best coverage of 15.79% on this task. For mixed ID/OOD, using models' softmax confidences for abstention decisions performs very poorly, answering <5% of questions at 1% risk of error even when faced with only 10% OOD examples, but a learned selection function with LYP can increase that to 25.38% C@1%.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2306.08751

PDF

https://arxiv.org/pdf/2306.08751

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4380993853

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4380993853

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2306.08751

Digital Object Identifier

Title

Improving Selective Visual Question Answering by Learning from Your Peers

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-06-14

Full publication date if available

Authors

Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, Marcus Rohrbach

List of authors in order

Landing page

https://arxiv.org/abs/2306.08751

Publisher landing page

PDF URL

https://arxiv.org/pdf/2306.08751

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2306.08751

Direct OA link when available

Concepts

Computer science, Question answering, Correctness, Machine learning, Metric (unit), Artificial intelligence, Adversarial system, Economics, Programming language, Operations management

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload