Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm Article Swipe

MAKE IMPACT

Jinrong Zhang , Penghui Wang , Chun‐Xiao Liu , Wei Liu , Dian Jin , Qiong Zhang , Erli Meng , Zhengnan Hu · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2412.10719

To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, \textbf{Image Prompt Paradigm}. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named \textbf{MI Grounding} for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2412.10719

PDF

https://arxiv.org/pdf/2412.10719

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4405468534

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4405468534

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2412.10719

Digital Object Identifier

Title

Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-12-14

Full publication date if available

Authors

Jinrong Zhang, Penghui Wang, Chun‐Xiao Liu, Wei Liu, Dian Jin, Qiong Zhang, Erli Meng, Zhengnan Hu

List of authors in order

Landing page

https://arxiv.org/abs/2412.10719

Publisher landing page

PDF URL

https://arxiv.org/pdf/2412.10719

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2412.10719

Direct OA link when available

Concepts

Perception, Set (abstract data type), Image (mathematics), Computer science, Cognitive psychology, Psychology, Computer vision, Neuroscience, Programming language

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload