Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks Article Swipe

MAKE IMPACT

Md Zarif Hossain , Ahmed Imteaj · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2409.07353

Large Vision-Language Models (LVLMs), trained on multimodal big datasets, have significantly advanced AI by excelling in vision-language tasks. However, these models remain vulnerable to adversarial attacks, particularly jailbreak attacks, which bypass safety protocols and cause the model to generate misleading or harmful responses. This vulnerability stems from both the inherent susceptibilities of LLMs and the expanded attack surface introduced by the visual modality. We propose Sim-CLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture. This approach maximizes cosine similarity between perturbed and clean samples, facilitating resilience against adversarial manipulations. Sim-CLIP+ offers a plug-and-play solution, allowing seamless integration into existing LVLM architectures as a robust vision encoder. Unlike previous defenses, our method requires no structural modifications to the LVLM and incurs minimal computational overhead. Sim-CLIP+ demonstrates effectiveness against both gradient-based adversarial attacks and various jailbreak techniques. We evaluate Sim-CLIP+ against three distinct jailbreak attack strategies and perform clean evaluations using standard downstream datasets, including COCO for image captioning and OKVQA for visual question answering. Extensive experiments demonstrate that Sim-CLIP+ maintains high clean accuracy while substantially improving robustness against both gradient-based adversarial attacks and jailbreak techniques. Our code and robust vision encoders are available at https://github.com/speedlab-git/Robust-Encoder-against-Jailbreak-attack.git.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2409.07353

PDF

https://arxiv.org/pdf/2409.07353

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4403621984

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4403621984

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2409.07353

Digital Object Identifier

Title

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-09-11

Full publication date if available

Authors

Md Zarif Hossain, Ahmed Imteaj

List of authors in order

Landing page

https://arxiv.org/abs/2409.07353

Publisher landing page

PDF URL

https://arxiv.org/pdf/2409.07353

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2409.07353

Direct OA link when available

Concepts

Adversarial system, Computer science, Encoder, Artificial intelligence, Computer security, Computer vision, Operating system

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload