FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models Article Swipe

MAKE IMPACT

Barbara Toniella Corradini , Mustafa Shukor , Paul Couairon , Guillaume Couairon , Franco Scarselli , Matthieu Cord · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.1109/ijcnn64981.2025.11227428

Foundation models have exhibited unprecedented capabilities in tackling many domains and tasks. Models such as CLIP are currently widely used to bridge cross-modal representations, and text-to-image diffusion models are arguably the leading models in terms of realistic image generation. Image generative models are trained on massive datasets that provide them with powerful internal spatial representations. In this work, we explore the potential benefits of such representations, beyond image generation, in particular, for dense visual prediction tasks. We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets, with pixel-level annotations. To avoid the annotation cost or training large diffusion models, we constraint our setup to be zero-shot and training-free. In a nutshell, our pipeline leverages different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. The pipeline is as follows: the image is passed to both a captioner model (i.e. BLIP) and a diffusion model (i.e., Stable Diffusion Model) to generate a text description and visual representation, respectively. The features are clustered and binarized to obtain class agnostic masks for each object. These masks are then mapped to a textual class, using the CLIP model to support open-vocabulary. Finally, we add a refinement step that allows to obtain a more precise segmentation mask. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets. In addition, we show very competitive results compared to the recent weakly-supervised segmentation approaches. We provide comprehensive experiments showing the superiority of diffusion model features compared to other pretrained models. Project page: https://bcorrad.github.io/freesegdiff/

Type

preprint

Language

en

Landing Page

https://doi.org/10.1109/ijcnn64981.2025.11227428

OA Status

green

References

37

Related Works

10

OpenAlex ID

https://openalex.org/W4393399541

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4393399541

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.1109/ijcnn64981.2025.11227428

Digital Object Identifier

Title

FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-06-30

Full publication date if available

Authors

Barbara Toniella Corradini, Mustafa Shukor, Paul Couairon, Guillaume Couairon, Franco Scarselli, Matthieu Cord

List of authors in order

Landing page

https://doi.org/10.1109/ijcnn64981.2025.11227428

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2403.20105

Direct OA link when available

Concepts

Training (meteorology), Vocabulary, Computer science, Diffusion, Segmentation, Artificial intelligence, Linguistics, Geography, Philosophy, Physics, Thermodynamics, Meteorology

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

References (count)

37

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload