Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting Article Swipe

MAKE IMPACT

Chen Huang , Skyler Seto , Hadi Pouransari , Mehrdad Farajtabar , Raviteja Vemulapalli , Fartash Faghri , Oncel Tuzel , Barry-John Theobald , Josh Susskind · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.24088

Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning performance. Knowledge is often preserved by matching the original and fine-tuned model weights or feature pairs. However, such point-wise matching can be too strong, without explicit awareness of the feature neighborhood structures that encode rich knowledge as well. We propose a novel regularization method Proxy-FDA that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further improved by informative proxies that are generated dynamically to increase data diversity. Experiments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find a strong correlation between forgetting and a distributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end, few-shot and continual tuning) and across different tasks like image classification, captioning and VQA.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2505.24088

PDF

https://arxiv.org/pdf/2505.24088

OA Status

green

OpenAlex ID

https://openalex.org/W4414856077

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4414856077

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2505.24088

Digital Object Identifier

Title

Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-05-30

Full publication date if available

Authors

Chen Huang, Skyler Seto, Hadi Pouransari, Mehrdad Farajtabar, Raviteja Vemulapalli, Fartash Faghri, Oncel Tuzel, Barry-John Theobald, Josh Susskind

List of authors in order

Landing page

https://arxiv.org/abs/2505.24088

Publisher landing page

PDF URL

https://arxiv.org/pdf/2505.24088

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2505.24088

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload