arXiv (Cornell University)
Understanding the Vulnerability of CLIP to Image Compression
November 2023 • Cangxiong Chen, Vinay P. Namboodiri, Julián Padget
CLIP is a widely used foundational vision-language model that is used for zero-shot image recognition and other image-text alignment tasks. We demonstrate that CLIP is vulnerable to change in image quality under compression. This surprising result is further analysed using an attribution method-Integrated Gradients. Using this attribution method, we are able to better understand both quantitatively and qualitatively exactly the nature in which the compression affects the zero-shot recognition accuracy of this mode…