ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images Article Swipe
Xiangtian Xue
,
Jiasong Wu
,
Youyong Kong
,
Lotfi Senhadji
,
Huazhong Shu
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2403.10004
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2403.10004
We present a novel image editing scenario termed Text-grounded Object Generation (TOG), defined as generating a new object in the real image spatially conditioned by textual descriptions. Existing diffusion models exhibit limitations of spatial perception in complex real-world scenes, relying on additional modalities to enforce constraints, and TOG imposes heightened challenges on scene comprehension under the weak supervision of linguistic information. We propose a universal framework ST-LDM based on Swin-Transformer, which can be integrated into any latent diffusion model with training-free backward guidance. ST-LDM encompasses a global-perceptual autoencoder with adaptable compression scales and hierarchical visual features, parallel with deformable multimodal transformer to generate region-wise guidance for the subsequent denoising process. We transcend the limitation of traditional attention mechanisms that only focus on existing visual features by introducing deformable feature alignment to hierarchically refine spatial positioning fused with multi-scale visual and linguistic information. Extensive Experiments demonstrate that our model enhances the localization of attention mechanisms while preserving the generative capabilities inherent to diffusion models.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2403.10004
- https://arxiv.org/pdf/2403.10004
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4392929748
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4392929748Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2403.10004Digital Object Identifier
- Title
-
ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real ImagesWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-03-15Full publication date if available
- Authors
-
Xiangtian Xue, Jiasong Wu, Youyong Kong, Lotfi Senhadji, Huazhong ShuList of authors in order
- Landing page
-
https://arxiv.org/abs/2403.10004Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2403.10004Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2403.10004Direct OA link when available
- Concepts
-
Object (grammar), Computer science, Object based, Grounded theory, Artificial intelligence, Natural language processing, Computer vision, Sociology, Qualitative research, Social scienceTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4392929748 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2403.10004 |
| ids.doi | https://doi.org/10.48550/arxiv.2403.10004 |
| ids.openalex | https://openalex.org/W4392929748 |
| fwci | |
| type | preprint |
| title | ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11714 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9894000291824341 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Multimodal Machine Learning Applications |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.954800009727478 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T10824 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9387999773025513 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Image Retrieval and Classification Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2781238097 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6084116101264954 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q175026 |
| concepts[0].display_name | Object (grammar) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.5033437609672546 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C3019973339 |
| concepts[2].level | 3 |
| concepts[2].score | 0.4260904788970947 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q899523 |
| concepts[2].display_name | Object based |
| concepts[3].id | https://openalex.org/C156325361 |
| concepts[3].level | 3 |
| concepts[3].score | 0.4236887991428375 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q1152864 |
| concepts[3].display_name | Grounded theory |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.35679590702056885 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C204321447 |
| concepts[5].level | 1 |
| concepts[5].score | 0.3453523516654968 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[5].display_name | Natural language processing |
| concepts[6].id | https://openalex.org/C31972630 |
| concepts[6].level | 1 |
| concepts[6].score | 0.3348245620727539 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q844240 |
| concepts[6].display_name | Computer vision |
| concepts[7].id | https://openalex.org/C144024400 |
| concepts[7].level | 0 |
| concepts[7].score | 0.1396763026714325 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q21201 |
| concepts[7].display_name | Sociology |
| concepts[8].id | https://openalex.org/C190248442 |
| concepts[8].level | 2 |
| concepts[8].score | 0.09440582990646362 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q839486 |
| concepts[8].display_name | Qualitative research |
| concepts[9].id | https://openalex.org/C36289849 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q34749 |
| concepts[9].display_name | Social science |
| keywords[0].id | https://openalex.org/keywords/object |
| keywords[0].score | 0.6084116101264954 |
| keywords[0].display_name | Object (grammar) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.5033437609672546 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/object-based |
| keywords[2].score | 0.4260904788970947 |
| keywords[2].display_name | Object based |
| keywords[3].id | https://openalex.org/keywords/grounded-theory |
| keywords[3].score | 0.4236887991428375 |
| keywords[3].display_name | Grounded theory |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.35679590702056885 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/natural-language-processing |
| keywords[5].score | 0.3453523516654968 |
| keywords[5].display_name | Natural language processing |
| keywords[6].id | https://openalex.org/keywords/computer-vision |
| keywords[6].score | 0.3348245620727539 |
| keywords[6].display_name | Computer vision |
| keywords[7].id | https://openalex.org/keywords/sociology |
| keywords[7].score | 0.1396763026714325 |
| keywords[7].display_name | Sociology |
| keywords[8].id | https://openalex.org/keywords/qualitative-research |
| keywords[8].score | 0.09440582990646362 |
| keywords[8].display_name | Qualitative research |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2403.10004 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2403.10004 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2403.10004 |
| locations[1].id | doi:10.48550/arxiv.2403.10004 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2403.10004 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5094168504 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Xiangtian Xue |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Xue, Xiangtian |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5078388244 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-7171-1318 |
| authorships[1].author.display_name | Jiasong Wu |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Wu, Jiasong |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5008751186 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-2095-8470 |
| authorships[2].author.display_name | Youyong Kong |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Kong, Youyong |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5028457159 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-9434-6341 |
| authorships[3].author.display_name | Lotfi Senhadji |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Senhadji, Lotfi |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5100687822 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-3833-7915 |
| authorships[4].author.display_name | Huazhong Shu |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Shu, Huazhong |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2403.10004 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11714 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9894000291824341 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Multimodal Machine Learning Applications |
| related_works | https://openalex.org/W2100786069, https://openalex.org/W4239112351, https://openalex.org/W4256166021, https://openalex.org/W2585146553, https://openalex.org/W2156352682, https://openalex.org/W4298625047, https://openalex.org/W1991139010, https://openalex.org/W4205448459, https://openalex.org/W3197862913, https://openalex.org/W2266572308 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2403.10004 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2403.10004 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2403.10004 |
| primary_location.id | pmh:oai:arXiv.org:2403.10004 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2403.10004 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2403.10004 |
| publication_date | 2024-03-15 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 2, 15, 63, 85 |
| abstract_inverted_index.We | 0, 61, 110 |
| abstract_inverted_index.as | 13 |
| abstract_inverted_index.be | 72 |
| abstract_inverted_index.by | 24, 125 |
| abstract_inverted_index.in | 18, 35 |
| abstract_inverted_index.of | 32, 58, 114, 151 |
| abstract_inverted_index.on | 40, 51, 68, 121 |
| abstract_inverted_index.to | 43, 101, 130, 160 |
| abstract_inverted_index.TOG | 47 |
| abstract_inverted_index.and | 46, 92, 139 |
| abstract_inverted_index.any | 75 |
| abstract_inverted_index.can | 71 |
| abstract_inverted_index.for | 105 |
| abstract_inverted_index.new | 16 |
| abstract_inverted_index.our | 146 |
| abstract_inverted_index.the | 19, 55, 106, 112, 149, 156 |
| abstract_inverted_index.into | 74 |
| abstract_inverted_index.only | 119 |
| abstract_inverted_index.real | 20 |
| abstract_inverted_index.that | 118, 145 |
| abstract_inverted_index.weak | 56 |
| abstract_inverted_index.with | 79, 88, 97, 136 |
| abstract_inverted_index.based | 67 |
| abstract_inverted_index.focus | 120 |
| abstract_inverted_index.fused | 135 |
| abstract_inverted_index.image | 4, 21 |
| abstract_inverted_index.model | 78, 147 |
| abstract_inverted_index.novel | 3 |
| abstract_inverted_index.scene | 52 |
| abstract_inverted_index.under | 54 |
| abstract_inverted_index.which | 70 |
| abstract_inverted_index.while | 154 |
| abstract_inverted_index.(TOG), | 11 |
| abstract_inverted_index.Object | 9 |
| abstract_inverted_index.ST-LDM | 66, 83 |
| abstract_inverted_index.latent | 76 |
| abstract_inverted_index.models | 29 |
| abstract_inverted_index.object | 17 |
| abstract_inverted_index.refine | 132 |
| abstract_inverted_index.scales | 91 |
| abstract_inverted_index.termed | 7 |
| abstract_inverted_index.visual | 94, 123, 138 |
| abstract_inverted_index.complex | 36 |
| abstract_inverted_index.defined | 12 |
| abstract_inverted_index.editing | 5 |
| abstract_inverted_index.enforce | 44 |
| abstract_inverted_index.exhibit | 30 |
| abstract_inverted_index.feature | 128 |
| abstract_inverted_index.imposes | 48 |
| abstract_inverted_index.models. | 162 |
| abstract_inverted_index.present | 1 |
| abstract_inverted_index.propose | 62 |
| abstract_inverted_index.relying | 39 |
| abstract_inverted_index.scenes, | 38 |
| abstract_inverted_index.spatial | 33, 133 |
| abstract_inverted_index.textual | 25 |
| abstract_inverted_index.Existing | 27 |
| abstract_inverted_index.backward | 81 |
| abstract_inverted_index.enhances | 148 |
| abstract_inverted_index.existing | 122 |
| abstract_inverted_index.features | 124 |
| abstract_inverted_index.generate | 102 |
| abstract_inverted_index.guidance | 104 |
| abstract_inverted_index.inherent | 159 |
| abstract_inverted_index.parallel | 96 |
| abstract_inverted_index.process. | 109 |
| abstract_inverted_index.scenario | 6 |
| abstract_inverted_index.Extensive | 142 |
| abstract_inverted_index.adaptable | 89 |
| abstract_inverted_index.alignment | 129 |
| abstract_inverted_index.attention | 116, 152 |
| abstract_inverted_index.denoising | 108 |
| abstract_inverted_index.diffusion | 28, 77, 161 |
| abstract_inverted_index.features, | 95 |
| abstract_inverted_index.framework | 65 |
| abstract_inverted_index.guidance. | 82 |
| abstract_inverted_index.spatially | 22 |
| abstract_inverted_index.transcend | 111 |
| abstract_inverted_index.universal | 64 |
| abstract_inverted_index.Generation | 10 |
| abstract_inverted_index.additional | 41 |
| abstract_inverted_index.challenges | 50 |
| abstract_inverted_index.deformable | 98, 127 |
| abstract_inverted_index.generating | 14 |
| abstract_inverted_index.generative | 157 |
| abstract_inverted_index.heightened | 49 |
| abstract_inverted_index.integrated | 73 |
| abstract_inverted_index.limitation | 113 |
| abstract_inverted_index.linguistic | 59, 140 |
| abstract_inverted_index.mechanisms | 117, 153 |
| abstract_inverted_index.modalities | 42 |
| abstract_inverted_index.multimodal | 99 |
| abstract_inverted_index.perception | 34 |
| abstract_inverted_index.preserving | 155 |
| abstract_inverted_index.real-world | 37 |
| abstract_inverted_index.subsequent | 107 |
| abstract_inverted_index.Experiments | 143 |
| abstract_inverted_index.autoencoder | 87 |
| abstract_inverted_index.compression | 90 |
| abstract_inverted_index.conditioned | 23 |
| abstract_inverted_index.demonstrate | 144 |
| abstract_inverted_index.encompasses | 84 |
| abstract_inverted_index.introducing | 126 |
| abstract_inverted_index.limitations | 31 |
| abstract_inverted_index.multi-scale | 137 |
| abstract_inverted_index.positioning | 134 |
| abstract_inverted_index.region-wise | 103 |
| abstract_inverted_index.supervision | 57 |
| abstract_inverted_index.traditional | 115 |
| abstract_inverted_index.transformer | 100 |
| abstract_inverted_index.capabilities | 158 |
| abstract_inverted_index.constraints, | 45 |
| abstract_inverted_index.hierarchical | 93 |
| abstract_inverted_index.information. | 60, 141 |
| abstract_inverted_index.localization | 150 |
| abstract_inverted_index.Text-grounded | 8 |
| abstract_inverted_index.comprehension | 53 |
| abstract_inverted_index.descriptions. | 26 |
| abstract_inverted_index.training-free | 80 |
| abstract_inverted_index.hierarchically | 131 |
| abstract_inverted_index.Swin-Transformer, | 69 |
| abstract_inverted_index.global-perceptual | 86 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |