Describe Anything: Detailed Localized Image and Video Captioning Article Swipe

PDF

Long Lian , Yifan Ding , Yunhao Ge , Sifei Liu , Hanzi Mao , Boyi Li , Marco Pavone , Mingyu Liu , Trevor Darrell , Adam Yala , Yin Cui ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2504.16072

Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

Related Topics

Truth And Reconciliation Commission Of Canada

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2504.16072
PDF: https://arxiv.org/pdf/2504.16072
OA Status: green
OpenAlex ID: https://openalex.org/W4414634784

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4414634784

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2504.16072

Digital Object Identifier
Title: Describe Anything: Detailed Localized Image and Video Captioning

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-04-22

Full publication date if available
Authors: Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Mingyu Liu, Trevor Darrell, Adam Yala, Yin Cui

List of authors in order
Landing page: https://arxiv.org/abs/2504.16072

Publisher landing page
PDF URL: https://arxiv.org/pdf/2504.16072

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2504.16072

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4414634784
doi	https://doi.org/10.48550/arxiv.2504.16072
ids.doi	https://doi.org/10.48550/arxiv.2504.16072
ids.openalex	https://openalex.org/W4414634784
fwci	0.0
type	preprint
title	Describe Anything: Detailed Localized Image and Video Captioning
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11714
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9998000264167786
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Multimodal Machine Learning Applications
topics[1].id	https://openalex.org/T11439
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.991599977016449
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1707
topics[1].subfield.display_name	Computer Vision and Pattern Recognition
topics[1].display_name	Video Analysis and Summarization
topics[2].id	https://openalex.org/T10627
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9911999702453613
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1707
topics[2].subfield.display_name	Computer Vision and Pattern Recognition
topics[2].display_name	Advanced Image and Video Retrieval Techniques
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2504.16072
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license	cc-by
locations[0].pdf_url	https://arxiv.org/pdf/2504.16072
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id	https://openalex.org/licenses/cc-by
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2504.16072
locations[1].id	doi:10.48550/arxiv.2504.16072
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2504.16072
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5114232525
authorships[0].author.orcid
authorships[0].author.display_name	Long Lian
authorships[0].author_position	first
authorships[0].raw_author_name	Lian, Long
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5101822673
authorships[1].author.orcid	https://orcid.org/0000-0003-1973-8374
authorships[1].author.display_name	Yifan Ding
authorships[1].author_position	middle
authorships[1].raw_author_name	Ding, Yifan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5038696465
authorships[2].author.orcid	https://orcid.org/0000-0002-8110-9280
authorships[2].author.display_name	Yunhao Ge
authorships[2].author_position	middle
authorships[2].raw_author_name	Ge, Yunhao
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5049815485
authorships[3].author.orcid	https://orcid.org/0000-0002-6011-3686
authorships[3].author.display_name	Sifei Liu
authorships[3].author_position	middle
authorships[3].raw_author_name	Liu, Sifei
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5021948346
authorships[4].author.orcid	https://orcid.org/0000-0002-2186-2991
authorships[4].author.display_name	Hanzi Mao
authorships[4].author_position	middle
authorships[4].raw_author_name	Mao, Hanzi
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5101884383
authorships[5].author.orcid	https://orcid.org/0000-0002-8921-3808
authorships[5].author.display_name	Boyi Li
authorships[5].author_position	middle
authorships[5].raw_author_name	Li, Boyi
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5050003000
authorships[6].author.orcid	https://orcid.org/0000-0002-0206-4337
authorships[6].author.display_name	Marco Pavone
authorships[6].author_position	middle
authorships[6].raw_author_name	Pavone, Marco
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5115596512
authorships[7].author.orcid	https://orcid.org/0000-0001-9905-4399
authorships[7].author.display_name	Mingyu Liu
authorships[7].author_position	middle
authorships[7].raw_author_name	Liu, Ming-Yu
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5029105520
authorships[8].author.orcid	https://orcid.org/0000-0001-5453-8533
authorships[8].author.display_name	Trevor Darrell
authorships[8].author_position	middle
authorships[8].raw_author_name	Darrell, Trevor
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5084801169
authorships[9].author.orcid	https://orcid.org/0000-0001-9576-2590
authorships[9].author.display_name	Adam Yala
authorships[9].author_position	middle
authorships[9].raw_author_name	Yala, Adam
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5100303262
authorships[10].author.orcid	https://orcid.org/0000-0003-2882-2033
authorships[10].author.display_name	Yin Cui
authorships[10].author_position	last
authorships[10].raw_author_name	Cui, Yin
authorships[10].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2504.16072
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Describe Anything: Detailed Localized Image and Video Captioning
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11714
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9998000264167786
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Multimodal Machine Learning Applications
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2504.16072
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license	cc-by
best_oa_location.pdf_url	https://arxiv.org/pdf/2504.16072
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id	https://openalex.org/licenses/cc-by
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2504.16072
primary_location.id	pmh:oai:arXiv.org:2504.16072
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license	cc-by
primary_location.pdf_url	https://arxiv.org/pdf/2504.16072
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id	https://openalex.org/licenses/cc-by
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2504.16072
publication_date	2025-04-22
publication_year	2025
referenced_works_count	0
abstract_inverted_index.7	119
abstract_inverted_index.a	13, 26, 46, 57, 79, 103
abstract_inverted_index.To	69
abstract_inverted_index.We	19, 100
abstract_inverted_index.in	8
abstract_inverted_index.of	53, 73
abstract_inverted_index.on	111, 118
abstract_inverted_index.to	94, 106
abstract_inverted_index.we	77
abstract_inverted_index.DAM	34, 114
abstract_inverted_index.DLC	75, 108
abstract_inverted_index.and	2, 10, 39, 56, 92, 124, 129
abstract_inverted_index.for	5, 16, 29
abstract_inverted_index.its	66
abstract_inverted_index.key	44
abstract_inverted_index.new	116
abstract_inverted_index.the	21, 71
abstract_inverted_index.two	43
abstract_inverted_index.web	96
abstract_inverted_index.Data	83
abstract_inverted_index.SSL.	99
abstract_inverted_index.both	36
abstract_inverted_index.sets	115
abstract_inverted_index.with	65, 88
abstract_inverted_index.Model	24
abstract_inverted_index.data,	76
abstract_inverted_index.focal	47
abstract_inverted_index.image	128
abstract_inverted_index.local	37
abstract_inverted_index.model	27
abstract_inverted_index.using	98
abstract_inverted_index.video	130
abstract_inverted_index.which	49, 61
abstract_inverted_index.(DAM),	25
abstract_inverted_index.(DLC).	33
abstract_inverted_index.global	40
abstract_inverted_index.images	9, 97
abstract_inverted_index.starts	87
abstract_inverted_index.tackle	70
abstract_inverted_index.videos	11
abstract_inverted_index.vision	59
abstract_inverted_index.DLC-SDP	86
abstract_inverted_index.broader	67
abstract_inverted_index.context	41
abstract_inverted_index.details	38
abstract_inverted_index.ensures	50
abstract_inverted_index.expands	93
abstract_inverted_index.models.	18
abstract_inverted_index.precise	63
abstract_inverted_index.prompt,	48
abstract_inverted_index.propose	78
abstract_inverted_index.regions	7
abstract_inverted_index.relying	110
abstract_inverted_index.remains	12
abstract_inverted_index.through	42
abstract_inverted_index.without	109
abstract_inverted_index.Anything	23
abstract_inverted_index.Describe	22
abstract_inverted_index.Pipeline	84
abstract_inverted_index.accurate	3
abstract_inverted_index.context.	68
abstract_inverted_index.datasets	91
abstract_inverted_index.designed	28, 105
abstract_inverted_index.detailed	1, 30, 125
abstract_inverted_index.encoding	52
abstract_inverted_index.evaluate	107
abstract_inverted_index.existing	89
abstract_inverted_index.learning	81
abstract_inverted_index.regions,	55
abstract_inverted_index.scarcity	72
abstract_inverted_index.spanning	121
abstract_inverted_index.specific	6
abstract_inverted_index.targeted	54
abstract_inverted_index.backbone,	60
abstract_inverted_index.benchmark	104
abstract_inverted_index.captions.	113
abstract_inverted_index.challenge	15
abstract_inverted_index.introduce	20, 101
abstract_inverted_index.localized	31, 58, 127
abstract_inverted_index.preserves	35
abstract_inverted_index.reference	112
abstract_inverted_index.unlabeled	95
abstract_inverted_index.(DLC-SDP).	85
abstract_inverted_index.DLC-Bench,	102
abstract_inverted_index.Generating	0
abstract_inverted_index.benchmarks	120
abstract_inverted_index.captioning	32
abstract_inverted_index.integrates	62
abstract_inverted_index.(SSL)-based	82
abstract_inverted_index.captioning.	131
abstract_inverted_index.fundamental	14
abstract_inverted_index.descriptions	4
abstract_inverted_index.high-quality	74
abstract_inverted_index.innovations:	45
abstract_inverted_index.localization	64
abstract_inverted_index.segmentation	90
abstract_inverted_index.phrase-level,	123
abstract_inverted_index.keyword-level,	122
abstract_inverted_index.multi-sentence	126
abstract_inverted_index.Semi-supervised	80
abstract_inverted_index.high-resolution	51
abstract_inverted_index.vision-language	17
abstract_inverted_index.state-of-the-art	117
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	11
citation_normalized_percentile