Reimagining Safety Alignment with An Image Article Swipe
Yifan Xia
,
Guorui Chen
,
Wenqian Yu
,
Zhijiang Li
,
Philip Torr
,
Jindong Gu
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2511.00509
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2511.00509
Large language models (LLMs) excel in diverse applications but face dual challenges: generating harmful content under jailbreak attacks and over-refusal of benign queries due to rigid safety mechanisms. These issues are further complicated by the need to accommodate different value systems and precisely align with given safety preferences. Moreover, traditional methods like SFT and RLHF lack this capability due to their costly parameter tuning requirements and inability to support multiple value systems within a single model. These problems are more obvious in multimodal large language models (MLLMs), especially in terms of heightened over-refusal in cross-modal tasks and new security risks arising from expanded attack surfaces. We propose Magic Image, an optimization-driven visual prompt framework that enhances security while reducing over-refusal. By optimizing image prompts using harmful/benign samples, our method enables a single model to adapt to different value systems and better align with given safety preferences without parameter updates. Experiments demonstrate improved safety-effectiveness balance across diverse datasets while preserving model performance, offering a practical solution for deployable MLLM safety alignment.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Landing Page
- http://arxiv.org/abs/2511.00509
- https://arxiv.org/pdf/2511.00509
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4415937999
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4415937999Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2511.00509Digital Object Identifier
- Title
-
Reimagining Safety Alignment with An ImageWork title
- Type
-
preprintOpenAlex work type
- Publication year
-
2025Year of publication
- Publication date
-
2025-11-01Full publication date if available
- Authors
-
Yifan Xia, Guorui Chen, Wenqian Yu, Zhijiang Li, Philip Torr, Jindong GuList of authors in order
- Landing page
-
https://arxiv.org/abs/2511.00509Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2511.00509Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2511.00509Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4415937999 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2511.00509 |
| ids.doi | https://doi.org/10.48550/arxiv.2511.00509 |
| ids.openalex | https://openalex.org/W4415937999 |
| fwci | |
| type | preprint |
| title | Reimagining Safety Alignment with An Image |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | |
| locations[0].id | pmh:oai:arXiv.org:2511.00509 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2511.00509 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2511.00509 |
| locations[1].id | doi:10.48550/arxiv.2511.00509 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2511.00509 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5109814269 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-3950-1617 |
| authorships[0].author.display_name | Yifan Xia |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Xia, Yifan |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5103268543 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Guorui Chen |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chen, Guorui |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5057799498 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-5956-5650 |
| authorships[2].author.display_name | Wenqian Yu |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Yu, Wenqian |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5061369238 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-9712-2220 |
| authorships[3].author.display_name | Zhijiang Li |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Li, Zhijiang |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5109615038 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Philip Torr |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Torr, Philip |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5055994909 |
| authorships[5].author.orcid | https://orcid.org/0009-0000-0574-0129 |
| authorships[5].author.display_name | Jindong Gu |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Gu, Jindong |
| authorships[5].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2511.00509 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-11-05T00:00:00 |
| display_name | Reimagining Safety Alignment with An Image |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-07T23:20:04.922697 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2511.00509 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2511.00509 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2511.00509 |
| primary_location.id | pmh:oai:arXiv.org:2511.00509 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2511.00509 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2511.00509 |
| publication_date | 2025-11-01 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 73, 130, 162 |
| abstract_inverted_index.By | 120 |
| abstract_inverted_index.We | 105 |
| abstract_inverted_index.an | 109 |
| abstract_inverted_index.by | 33 |
| abstract_inverted_index.in | 5, 81, 88, 93 |
| abstract_inverted_index.of | 20, 90 |
| abstract_inverted_index.to | 24, 36, 59, 67, 133, 135 |
| abstract_inverted_index.SFT | 52 |
| abstract_inverted_index.and | 18, 41, 53, 65, 96, 139 |
| abstract_inverted_index.are | 30, 78 |
| abstract_inverted_index.but | 8 |
| abstract_inverted_index.due | 23, 58 |
| abstract_inverted_index.for | 165 |
| abstract_inverted_index.new | 97 |
| abstract_inverted_index.our | 127 |
| abstract_inverted_index.the | 34 |
| abstract_inverted_index.MLLM | 167 |
| abstract_inverted_index.RLHF | 54 |
| abstract_inverted_index.dual | 10 |
| abstract_inverted_index.face | 9 |
| abstract_inverted_index.from | 101 |
| abstract_inverted_index.lack | 55 |
| abstract_inverted_index.like | 51 |
| abstract_inverted_index.more | 79 |
| abstract_inverted_index.need | 35 |
| abstract_inverted_index.that | 114 |
| abstract_inverted_index.this | 56 |
| abstract_inverted_index.with | 44, 142 |
| abstract_inverted_index.Large | 0 |
| abstract_inverted_index.Magic | 107 |
| abstract_inverted_index.These | 28, 76 |
| abstract_inverted_index.adapt | 134 |
| abstract_inverted_index.align | 43, 141 |
| abstract_inverted_index.excel | 4 |
| abstract_inverted_index.given | 45, 143 |
| abstract_inverted_index.image | 122 |
| abstract_inverted_index.large | 83 |
| abstract_inverted_index.model | 132, 159 |
| abstract_inverted_index.rigid | 25 |
| abstract_inverted_index.risks | 99 |
| abstract_inverted_index.tasks | 95 |
| abstract_inverted_index.terms | 89 |
| abstract_inverted_index.their | 60 |
| abstract_inverted_index.under | 15 |
| abstract_inverted_index.using | 124 |
| abstract_inverted_index.value | 39, 70, 137 |
| abstract_inverted_index.while | 117, 157 |
| abstract_inverted_index.(LLMs) | 3 |
| abstract_inverted_index.Image, | 108 |
| abstract_inverted_index.across | 154 |
| abstract_inverted_index.attack | 103 |
| abstract_inverted_index.benign | 21 |
| abstract_inverted_index.better | 140 |
| abstract_inverted_index.costly | 61 |
| abstract_inverted_index.issues | 29 |
| abstract_inverted_index.method | 128 |
| abstract_inverted_index.model. | 75 |
| abstract_inverted_index.models | 2, 85 |
| abstract_inverted_index.prompt | 112 |
| abstract_inverted_index.safety | 26, 46, 144, 168 |
| abstract_inverted_index.single | 74, 131 |
| abstract_inverted_index.tuning | 63 |
| abstract_inverted_index.visual | 111 |
| abstract_inverted_index.within | 72 |
| abstract_inverted_index.arising | 100 |
| abstract_inverted_index.attacks | 17 |
| abstract_inverted_index.balance | 153 |
| abstract_inverted_index.content | 14 |
| abstract_inverted_index.diverse | 6, 155 |
| abstract_inverted_index.enables | 129 |
| abstract_inverted_index.further | 31 |
| abstract_inverted_index.harmful | 13 |
| abstract_inverted_index.methods | 50 |
| abstract_inverted_index.obvious | 80 |
| abstract_inverted_index.prompts | 123 |
| abstract_inverted_index.propose | 106 |
| abstract_inverted_index.queries | 22 |
| abstract_inverted_index.support | 68 |
| abstract_inverted_index.systems | 40, 71, 138 |
| abstract_inverted_index.without | 146 |
| abstract_inverted_index.(MLLMs), | 86 |
| abstract_inverted_index.datasets | 156 |
| abstract_inverted_index.enhances | 115 |
| abstract_inverted_index.expanded | 102 |
| abstract_inverted_index.improved | 151 |
| abstract_inverted_index.language | 1, 84 |
| abstract_inverted_index.multiple | 69 |
| abstract_inverted_index.offering | 161 |
| abstract_inverted_index.problems | 77 |
| abstract_inverted_index.reducing | 118 |
| abstract_inverted_index.samples, | 126 |
| abstract_inverted_index.security | 98, 116 |
| abstract_inverted_index.solution | 164 |
| abstract_inverted_index.updates. | 148 |
| abstract_inverted_index.Moreover, | 48 |
| abstract_inverted_index.different | 38, 136 |
| abstract_inverted_index.framework | 113 |
| abstract_inverted_index.inability | 66 |
| abstract_inverted_index.jailbreak | 16 |
| abstract_inverted_index.parameter | 62, 147 |
| abstract_inverted_index.practical | 163 |
| abstract_inverted_index.precisely | 42 |
| abstract_inverted_index.surfaces. | 104 |
| abstract_inverted_index.alignment. | 169 |
| abstract_inverted_index.capability | 57 |
| abstract_inverted_index.deployable | 166 |
| abstract_inverted_index.especially | 87 |
| abstract_inverted_index.generating | 12 |
| abstract_inverted_index.heightened | 91 |
| abstract_inverted_index.multimodal | 82 |
| abstract_inverted_index.optimizing | 121 |
| abstract_inverted_index.preserving | 158 |
| abstract_inverted_index.Experiments | 149 |
| abstract_inverted_index.accommodate | 37 |
| abstract_inverted_index.challenges: | 11 |
| abstract_inverted_index.complicated | 32 |
| abstract_inverted_index.cross-modal | 94 |
| abstract_inverted_index.demonstrate | 150 |
| abstract_inverted_index.mechanisms. | 27 |
| abstract_inverted_index.preferences | 145 |
| abstract_inverted_index.traditional | 49 |
| abstract_inverted_index.applications | 7 |
| abstract_inverted_index.over-refusal | 19, 92 |
| abstract_inverted_index.performance, | 160 |
| abstract_inverted_index.preferences. | 47 |
| abstract_inverted_index.requirements | 64 |
| abstract_inverted_index.over-refusal. | 119 |
| abstract_inverted_index.harmful/benign | 125 |
| abstract_inverted_index.optimization-driven | 110 |
| abstract_inverted_index.safety-effectiveness | 152 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |