Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency Article Swipe
Michael Kirchhof
,
James Thornton
,
Pierre Ablin
,
Louis Béthune
,
Eugène Ndiaye
,
Marco Cuturi
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.06025
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.06025
The adoption of text-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration, fairness, or compute efficiency. We focus in this work on two issues that arise when deploying these models: a lack of diversity when prompting images, and a tendency to recreate images from the training set. To solve both problems, we propose a method that coaxes the sampled trajectories of pretrained diffusion models to land on images that fall outside of a reference set. We achieve this by adding repellency terms to the diffusion SDE throughout the generation trajectory, which are triggered whenever the path is expected to land too closely to an image in the shielded reference set. Our method is sparse in the sense that these repellency terms are zero and inactive most of the time, and even more so towards the end of the generation trajectory. Our method, named SPELL for sparse repellency, can be used either with a static reference set that contains protected images, or dynamically, by updating the set at each timestep with the expected images concurrently generated within a batch, and with the images of previously generated batches. We show that adding SPELL to popular diffusion models improves their diversity while impacting their FID only marginally, and performs comparatively better than other recent training-free diversity methods. We also demonstrate how SPELL can ensure a shielded generation away from a very large set of protected images by considering all 1.2M images from ImageNet as the protected set.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2410.06025
- https://arxiv.org/pdf/2410.06025
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403363154
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403363154Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2410.06025Digital Object Identifier
- Title
-
Shielded Diffusion: Generating Novel and Diverse Images using Sparse RepellencyWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-10-08Full publication date if available
- Authors
-
Michael Kirchhof, James Thornton, Pierre Ablin, Louis Béthune, Eugène Ndiaye, Marco CuturiList of authors in order
- Landing page
-
https://arxiv.org/abs/2410.06025Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2410.06025Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2410.06025Direct OA link when available
- Concepts
-
Shielded cable, Image (mathematics), Diffusion, Computer science, Artificial intelligence, Physics, Thermodynamics, TelecommunicationsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403363154 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2410.06025 |
| ids.doi | https://doi.org/10.48550/arxiv.2410.06025 |
| ids.openalex | https://openalex.org/W4403363154 |
| fwci | |
| type | preprint |
| title | Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T13910 |
| topics[0].field.id | https://openalex.org/fields/33 |
| topics[0].field.display_name | Social Sciences |
| topics[0].score | 0.8019999861717224 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/3300 |
| topics[0].subfield.display_name | General Social Sciences |
| topics[0].display_name | Computational and Text Analysis Methods |
| topics[1].id | https://openalex.org/T10597 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.7146000266075134 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2202 |
| topics[1].subfield.display_name | Aerospace Engineering |
| topics[1].display_name | Nuclear reactor physics and engineering |
| topics[2].id | https://openalex.org/T11242 |
| topics[2].field.id | https://openalex.org/fields/25 |
| topics[2].field.display_name | Materials Science |
| topics[2].score | 0.6784999966621399 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2505 |
| topics[2].subfield.display_name | Materials Chemistry |
| topics[2].display_name | Nuclear Materials and Properties |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C77590175 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8195538520812988 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q3506009 |
| concepts[0].display_name | Shielded cable |
| concepts[1].id | https://openalex.org/C115961682 |
| concepts[1].level | 2 |
| concepts[1].score | 0.574660062789917 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q860623 |
| concepts[1].display_name | Image (mathematics) |
| concepts[2].id | https://openalex.org/C69357855 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5627022385597229 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q163214 |
| concepts[2].display_name | Diffusion |
| concepts[3].id | https://openalex.org/C41008148 |
| concepts[3].level | 0 |
| concepts[3].score | 0.4530850648880005 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[3].display_name | Computer science |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.38524389266967773 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C121332964 |
| concepts[5].level | 0 |
| concepts[5].score | 0.3151690363883972 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[5].display_name | Physics |
| concepts[6].id | https://openalex.org/C97355855 |
| concepts[6].level | 1 |
| concepts[6].score | 0.12425115704536438 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11473 |
| concepts[6].display_name | Thermodynamics |
| concepts[7].id | https://openalex.org/C76155785 |
| concepts[7].level | 1 |
| concepts[7].score | 0.09589278697967529 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q418 |
| concepts[7].display_name | Telecommunications |
| keywords[0].id | https://openalex.org/keywords/shielded-cable |
| keywords[0].score | 0.8195538520812988 |
| keywords[0].display_name | Shielded cable |
| keywords[1].id | https://openalex.org/keywords/image |
| keywords[1].score | 0.574660062789917 |
| keywords[1].display_name | Image (mathematics) |
| keywords[2].id | https://openalex.org/keywords/diffusion |
| keywords[2].score | 0.5627022385597229 |
| keywords[2].display_name | Diffusion |
| keywords[3].id | https://openalex.org/keywords/computer-science |
| keywords[3].score | 0.4530850648880005 |
| keywords[3].display_name | Computer science |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.38524389266967773 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/physics |
| keywords[5].score | 0.3151690363883972 |
| keywords[5].display_name | Physics |
| keywords[6].id | https://openalex.org/keywords/thermodynamics |
| keywords[6].score | 0.12425115704536438 |
| keywords[6].display_name | Thermodynamics |
| keywords[7].id | https://openalex.org/keywords/telecommunications |
| keywords[7].score | 0.09589278697967529 |
| keywords[7].display_name | Telecommunications |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2410.06025 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2410.06025 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2410.06025 |
| locations[1].id | doi:10.48550/arxiv.2410.06025 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2410.06025 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5035574656 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-4521-9391 |
| authorships[0].author.display_name | Michael Kirchhof |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Kirchhof, Michael |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5109957064 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | James Thornton |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Thornton, James |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5042340163 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-4277-5202 |
| authorships[2].author.display_name | Pierre Ablin |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Ablin, Pierre |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5060903205 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-1498-8251 |
| authorships[3].author.display_name | Louis Béthune |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Béthune, Louis |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5021079415 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-4898-434X |
| authorships[4].author.display_name | Eugène Ndiaye |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Ndiaye, Eugene |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5017263427 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-1934-0588 |
| authorships[5].author.display_name | Marco Cuturi |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Cuturi, Marco |
| authorships[5].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2410.06025 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-10-13T00:00:00 |
| display_name | Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T13910 |
| primary_topic.field.id | https://openalex.org/fields/33 |
| primary_topic.field.display_name | Social Sciences |
| primary_topic.score | 0.8019999861717224 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/3300 |
| primary_topic.subfield.display_name | General Social Sciences |
| primary_topic.display_name | Computational and Text Analysis Methods |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2935759653, https://openalex.org/W3105167352, https://openalex.org/W54078636, https://openalex.org/W2954470139, https://openalex.org/W1501425562, https://openalex.org/W2902782467, https://openalex.org/W3084825885, https://openalex.org/W2298861036, https://openalex.org/W2271181815 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2410.06025 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2410.06025 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2410.06025 |
| primary_location.id | pmh:oai:arXiv.org:2410.06025 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2410.06025 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2410.06025 |
| publication_date | 2024-10-08 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 38, 46, 61, 80, 160, 184, 229, 234 |
| abstract_inverted_index.To | 55 |
| abstract_inverted_index.We | 24, 83, 194, 222 |
| abstract_inverted_index.an | 111 |
| abstract_inverted_index.as | 248 |
| abstract_inverted_index.at | 174 |
| abstract_inverted_index.be | 156 |
| abstract_inverted_index.by | 86, 170, 241 |
| abstract_inverted_index.in | 26, 113, 122 |
| abstract_inverted_index.is | 104, 120 |
| abstract_inverted_index.of | 2, 15, 40, 68, 79, 134, 144, 190, 238 |
| abstract_inverted_index.on | 29, 74 |
| abstract_inverted_index.or | 21, 168 |
| abstract_inverted_index.so | 140 |
| abstract_inverted_index.to | 48, 72, 90, 106, 110, 199 |
| abstract_inverted_index.we | 59 |
| abstract_inverted_index.FID | 209 |
| abstract_inverted_index.Our | 118, 148 |
| abstract_inverted_index.SDE | 93 |
| abstract_inverted_index.The | 0 |
| abstract_inverted_index.all | 243 |
| abstract_inverted_index.and | 45, 131, 137, 186, 212 |
| abstract_inverted_index.are | 99, 129 |
| abstract_inverted_index.can | 155, 227 |
| abstract_inverted_index.end | 143 |
| abstract_inverted_index.for | 152 |
| abstract_inverted_index.how | 225 |
| abstract_inverted_index.set | 163, 173, 237 |
| abstract_inverted_index.the | 13, 52, 65, 91, 95, 102, 114, 123, 135, 142, 145, 172, 178, 188, 249 |
| abstract_inverted_index.too | 108 |
| abstract_inverted_index.two | 30 |
| abstract_inverted_index.1.2M | 244 |
| abstract_inverted_index.also | 223 |
| abstract_inverted_index.away | 232 |
| abstract_inverted_index.both | 57 |
| abstract_inverted_index.each | 175 |
| abstract_inverted_index.even | 138 |
| abstract_inverted_index.fall | 77 |
| abstract_inverted_index.from | 51, 233, 246 |
| abstract_inverted_index.lack | 39 |
| abstract_inverted_index.land | 73, 107 |
| abstract_inverted_index.lens | 14 |
| abstract_inverted_index.like | 18 |
| abstract_inverted_index.more | 139 |
| abstract_inverted_index.most | 133 |
| abstract_inverted_index.only | 210 |
| abstract_inverted_index.over | 8 |
| abstract_inverted_index.path | 103 |
| abstract_inverted_index.set. | 54, 82, 117, 251 |
| abstract_inverted_index.show | 195 |
| abstract_inverted_index.than | 216 |
| abstract_inverted_index.that | 32, 63, 76, 125, 164, 196 |
| abstract_inverted_index.this | 27, 85 |
| abstract_inverted_index.used | 157 |
| abstract_inverted_index.very | 235 |
| abstract_inverted_index.when | 34, 42 |
| abstract_inverted_index.with | 159, 177, 187 |
| abstract_inverted_index.work | 28 |
| abstract_inverted_index.zero | 130 |
| abstract_inverted_index.SPELL | 151, 198, 226 |
| abstract_inverted_index.arise | 33 |
| abstract_inverted_index.focus | 25 |
| abstract_inverted_index.image | 112 |
| abstract_inverted_index.large | 236 |
| abstract_inverted_index.named | 150 |
| abstract_inverted_index.other | 217 |
| abstract_inverted_index.sense | 124 |
| abstract_inverted_index.solve | 56 |
| abstract_inverted_index.terms | 89, 128 |
| abstract_inverted_index.their | 204, 208 |
| abstract_inverted_index.these | 36, 126 |
| abstract_inverted_index.time, | 136 |
| abstract_inverted_index.under | 12 |
| abstract_inverted_index.which | 98 |
| abstract_inverted_index.while | 206 |
| abstract_inverted_index.adding | 87, 197 |
| abstract_inverted_index.batch, | 185 |
| abstract_inverted_index.better | 215 |
| abstract_inverted_index.coaxes | 64 |
| abstract_inverted_index.either | 158 |
| abstract_inverted_index.ensure | 228 |
| abstract_inverted_index.images | 50, 75, 180, 189, 240, 245 |
| abstract_inverted_index.issues | 31 |
| abstract_inverted_index.method | 62, 119 |
| abstract_inverted_index.models | 5, 71, 202 |
| abstract_inverted_index.raises | 6 |
| abstract_inverted_index.recent | 218 |
| abstract_inverted_index.sparse | 121, 153 |
| abstract_inverted_index.static | 161 |
| abstract_inverted_index.within | 183 |
| abstract_inverted_index.achieve | 84 |
| abstract_inverted_index.closely | 109 |
| abstract_inverted_index.compute | 22 |
| abstract_inverted_index.drawing | 10 |
| abstract_inverted_index.images, | 44, 167 |
| abstract_inverted_index.method, | 149 |
| abstract_inverted_index.metrics | 17 |
| abstract_inverted_index.models: | 37 |
| abstract_inverted_index.outside | 78 |
| abstract_inverted_index.popular | 200 |
| abstract_inverted_index.propose | 60 |
| abstract_inverted_index.sampled | 66 |
| abstract_inverted_index.towards | 141 |
| abstract_inverted_index.various | 16 |
| abstract_inverted_index.ImageNet | 247 |
| abstract_inverted_index.adoption | 1 |
| abstract_inverted_index.batches. | 193 |
| abstract_inverted_index.concerns | 7 |
| abstract_inverted_index.contains | 165 |
| abstract_inverted_index.expected | 105, 179 |
| abstract_inverted_index.improves | 203 |
| abstract_inverted_index.inactive | 132 |
| abstract_inverted_index.methods. | 221 |
| abstract_inverted_index.performs | 213 |
| abstract_inverted_index.recreate | 49 |
| abstract_inverted_index.scrutiny | 11 |
| abstract_inverted_index.shielded | 115, 230 |
| abstract_inverted_index.tendency | 47 |
| abstract_inverted_index.timestep | 176 |
| abstract_inverted_index.training | 53 |
| abstract_inverted_index.updating | 171 |
| abstract_inverted_index.whenever | 101 |
| abstract_inverted_index.deploying | 35 |
| abstract_inverted_index.diffusion | 4, 70, 92, 201 |
| abstract_inverted_index.diversity | 41, 205, 220 |
| abstract_inverted_index.fairness, | 20 |
| abstract_inverted_index.generated | 182, 192 |
| abstract_inverted_index.impacting | 207 |
| abstract_inverted_index.problems, | 58 |
| abstract_inverted_index.prompting | 43 |
| abstract_inverted_index.protected | 166, 239, 250 |
| abstract_inverted_index.reference | 81, 116, 162 |
| abstract_inverted_index.triggered | 100 |
| abstract_inverted_index.generation | 96, 146, 231 |
| abstract_inverted_index.pretrained | 69 |
| abstract_inverted_index.previously | 191 |
| abstract_inverted_index.repellency | 88, 127 |
| abstract_inverted_index.throughout | 94 |
| abstract_inverted_index.considering | 242 |
| abstract_inverted_index.demonstrate | 224 |
| abstract_inverted_index.efficiency. | 23 |
| abstract_inverted_index.marginally, | 211 |
| abstract_inverted_index.repellency, | 154 |
| abstract_inverted_index.trajectory, | 97 |
| abstract_inverted_index.trajectory. | 147 |
| abstract_inverted_index.calibration, | 19 |
| abstract_inverted_index.concurrently | 181 |
| abstract_inverted_index.dynamically, | 169 |
| abstract_inverted_index.reliability, | 9 |
| abstract_inverted_index.trajectories | 67 |
| abstract_inverted_index.comparatively | 214 |
| abstract_inverted_index.text-to-image | 3 |
| abstract_inverted_index.training-free | 219 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |