Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models Article Swipe

MAKE IMPACT

Song Fei , Li Yi , Wang Rui , Zhou, Jiahuan , Zheng, Changwen , Li, Jiangmeng · YOU? · · 2025 · Open Access ·

Test-time prompt tuning for vision-language models has demonstrated impressive generalization capabilities under zero-shot settings. However, tuning the learnable prompts solely based on unlabeled test data may induce prompt optimization bias, ultimately leading to suboptimal performance on downstream tasks. In this work, we analyze the underlying causes of prompt optimization bias from both the model and data perspectives. In terms of the model, the entropy minimization objective typically focuses on reducing the entropy of model predictions while overlooking their correctness. This can result in overconfident yet incorrect outputs, thereby compromising the quality of prompt optimization. On the data side, prompts affected by optimization bias can introduce misalignment between visual and textual modalities, which further aggravates the prompt optimization bias. To this end, we propose a Doubly Debiased Test-Time Prompt Tuning method. Specifically, we first introduce a dynamic retrieval-augmented modulation module that retrieves high-confidence knowledge from a dynamic knowledge base using the test image feature as a query, and uses the retrieved knowledge to modulate the predictions. Guided by the refined predictions, we further develop a reliability-aware prompt optimization module that incorporates a confidence-based weighted ensemble and cross-modal consistency distillation to impose regularization constraints during prompt tuning. Extensive experiments across 15 benchmark datasets involving both natural distribution shifts and cross-datasets generalization demonstrate that our method outperforms baselines, validating its effectiveness in mitigating prompt optimization bias.

Type

article

Landing Page

http://arxiv.org/abs/2511.11690

PDF

https://arxiv.org/pdf/2511.11690

OA Status

green

OpenAlex ID

https://openalex.org/W7106094599

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W7106094599

Canonical identifier for this work in OpenAlex

Title

Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models

Work title

Type

article

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-12

Full publication date if available

Authors

Song Fei, Li Yi, Wang Rui, Zhou, Jiahuan, Zheng, Changwen, Li, Jiangmeng

List of authors in order

Landing page

https://arxiv.org/abs/2511.11690

Publisher landing page

PDF URL

https://arxiv.org/pdf/2511.11690

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2511.11690

Direct OA link when available

Concepts

Computer science, Minification, Entropy (arrow of time), Artificial intelligence, Optimization problem, Generalization, Machine learning, Consistency (knowledge bases), Regularization (linguistics), Test data, Benchmark (surveying), Optimization algorithm, Synthetic data, Feature (linguistics), Data modeling, Cross entropy, Kullback–Leibler divergence, Algorithm, Data quality

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Full payload