Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration Article Swipe

PDF

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.22819

Vision-language models (VLM) have demonstrated impressive performance in image recognition by leveraging self-supervised training on large datasets. Their performance can be further improved by adapting to the test sample using test-time prompt tuning (TPT). Unfortunately, the singular focus of TPT approaches on improving the accuracy suffers from tunnel vision, and leads to degradation in confidence calibration. This limits the applicability of TPT in critical applications. We make three contributions in this work. (1) We posit that random or naive initialization of prompts leads to overfitting on a particular test sample, and is the main reason for miscalibration of the VLM after TPT. To mitigate the problem, we propose careful initialization of test time prompt using prior knowledge about the target label attributes from a large language model (LLM); (2) To further maintain the quality of prompts during \tpt, we propose a novel regularization loss to reduce intraclass distance, and increase inter-class distance between the learnt Through extensive experiments on different CLIP architectures and 15 datasets, we show that our approach can effectively improve the calibration after TPT. We report an average expected calibration error (ECE) of 4.11 with our method, TCA, compared to 11.7 for vanilla TPT, 6.12 for C-TPT (ICLR'24), 6.78 for DiffTPT (CVPR'23), and 8.43 for PromptAlign (NeurIPS'23). The code is publicly accessible at: https://github.com/rhebbalaguppe/TCA_PromptWithoutPanic.

Related Topics

Not Without My Daughter (Film)

Panic! At The Disco

Year Without A Summer

Panic Room

Rebel Without A Cause

Widespread Panic

Panic Attack

Men Without Hats

Not Without My Daughter (Book)

Without A Trace

Panic Disorder

Translators Without Borders

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2506.22819
PDF: https://arxiv.org/pdf/2506.22819
OA Status: green
OpenAlex ID: https://openalex.org/W4416507476

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4416507476

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2506.22819

Digital Object Identifier
Title: Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-06-28

Full publication date if available
Authors: Tamoghno Kandar, Chetan Arora

List of authors in order
Landing page: https://arxiv.org/abs/2506.22819

Publisher landing page
PDF URL: https://arxiv.org/pdf/2506.22819

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2506.22819

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4416507476
doi	https://doi.org/10.48550/arxiv.2506.22819
ids.doi	https://doi.org/10.48550/arxiv.2506.22819
ids.openalex	https://openalex.org/W4416507476
fwci
type	preprint
title	Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2506.22819
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2506.22819
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2506.22819
locations[1].id	doi:10.48550/arxiv.2506.22819
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2506.22819
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5120419412
authorships[0].author.orcid
authorships[0].author.display_name	Tamoghno Kandar
authorships[0].author_position	last
authorships[0].raw_author_name	Kandar, Tamoghno
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5019739552
authorships[1].author.orcid	https://orcid.org/0000-0003-1466-7386
authorships[1].author.display_name	Chetan Arora
authorships[1].author_position	middle
authorships[1].raw_author_name	Arora, Chetan
authorships[1].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2506.22819
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
has_fulltext	False
is_retracted	False
updated_date	2025-11-28T16:04:01.189229
primary_topic
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2506.22819
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2506.22819
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2506.22819
primary_location.id	pmh:oai:arXiv.org:2506.22819
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2506.22819
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2506.22819
publication_date	2025-06-28
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	86, 123, 140
abstract_inverted_index.15	163
abstract_inverted_index.To	102, 129
abstract_inverted_index.We	65, 73, 177
abstract_inverted_index.an	179
abstract_inverted_index.be	20
abstract_inverted_index.by	10, 23
abstract_inverted_index.in	7, 53, 62, 69
abstract_inverted_index.is	91, 212
abstract_inverted_index.of	38, 60, 80, 97, 110, 134, 185
abstract_inverted_index.on	14, 41, 85, 158
abstract_inverted_index.or	77
abstract_inverted_index.to	25, 51, 83, 144, 192
abstract_inverted_index.we	106, 138, 165
abstract_inverted_index.(1)	72
abstract_inverted_index.(2)	128
abstract_inverted_index.TPT	39, 61
abstract_inverted_index.The	210
abstract_inverted_index.VLM	99
abstract_inverted_index.and	49, 90, 148, 162, 205
abstract_inverted_index.at:	215
abstract_inverted_index.can	19, 170
abstract_inverted_index.for	95, 194, 198, 202, 207
abstract_inverted_index.our	168, 188
abstract_inverted_index.the	26, 35, 43, 58, 92, 98, 104, 118, 132, 153, 173
abstract_inverted_index.11.7	193
abstract_inverted_index.4.11	186
abstract_inverted_index.6.12	197
abstract_inverted_index.6.78	201
abstract_inverted_index.8.43	206
abstract_inverted_index.CLIP	160
abstract_inverted_index.TCA,	190
abstract_inverted_index.TPT,	196
abstract_inverted_index.TPT.	101, 176
abstract_inverted_index.This	56
abstract_inverted_index.code	211
abstract_inverted_index.from	46, 122
abstract_inverted_index.have	3
abstract_inverted_index.loss	143
abstract_inverted_index.main	93
abstract_inverted_index.make	66
abstract_inverted_index.show	166
abstract_inverted_index.test	27, 88, 111
abstract_inverted_index.that	75, 167
abstract_inverted_index.this	70
abstract_inverted_index.time	112
abstract_inverted_index.with	187
abstract_inverted_index.(ECE)	184
abstract_inverted_index.(VLM)	2
abstract_inverted_index.C-TPT	199
abstract_inverted_index.Their	17
abstract_inverted_index.\tpt,	137
abstract_inverted_index.about	117
abstract_inverted_index.after	100, 175
abstract_inverted_index.error	183
abstract_inverted_index.focus	37
abstract_inverted_index.image	8
abstract_inverted_index.label	120
abstract_inverted_index.large	15, 124
abstract_inverted_index.leads	50, 82
abstract_inverted_index.model	126
abstract_inverted_index.naive	78
abstract_inverted_index.novel	141
abstract_inverted_index.posit	74
abstract_inverted_index.prior	115
abstract_inverted_index.three	67
abstract_inverted_index.using	29, 114
abstract_inverted_index.work.	71
abstract_inverted_index.(LLM);	127
abstract_inverted_index.(TPT).	33
abstract_inverted_index.during	136
abstract_inverted_index.learnt	154
abstract_inverted_index.limits	57
abstract_inverted_index.models	1
abstract_inverted_index.prompt	31, 113
abstract_inverted_index.random	76
abstract_inverted_index.reason	94
abstract_inverted_index.reduce	145
abstract_inverted_index.report	178
abstract_inverted_index.sample	28
abstract_inverted_index.target	119
abstract_inverted_index.tuning	32
abstract_inverted_index.tunnel	47
abstract_inverted_index.DiffTPT	203
abstract_inverted_index.Through	155
abstract_inverted_index.average	180
abstract_inverted_index.between	152
abstract_inverted_index.careful	108
abstract_inverted_index.further	21, 130
abstract_inverted_index.improve	172
abstract_inverted_index.method,	189
abstract_inverted_index.prompts	81, 135
abstract_inverted_index.propose	107, 139
abstract_inverted_index.quality	133
abstract_inverted_index.sample,	89
abstract_inverted_index.suffers	45
abstract_inverted_index.vanilla	195
abstract_inverted_index.vision,	48
abstract_inverted_index.accuracy	44
abstract_inverted_index.adapting	24
abstract_inverted_index.approach	169
abstract_inverted_index.compared	191
abstract_inverted_index.critical	63
abstract_inverted_index.distance	151
abstract_inverted_index.expected	181
abstract_inverted_index.improved	22
abstract_inverted_index.increase	149
abstract_inverted_index.language	125
abstract_inverted_index.maintain	131
abstract_inverted_index.mitigate	103
abstract_inverted_index.problem,	105
abstract_inverted_index.publicly	213
abstract_inverted_index.singular	36
abstract_inverted_index.training	13
abstract_inverted_index.datasets,	164
abstract_inverted_index.datasets.	16
abstract_inverted_index.different	159
abstract_inverted_index.distance,	147
abstract_inverted_index.extensive	156
abstract_inverted_index.improving	42
abstract_inverted_index.knowledge	116
abstract_inverted_index.test-time	30
abstract_inverted_index.(CVPR'23),	204
abstract_inverted_index.(ICLR'24),	200
abstract_inverted_index.accessible	214
abstract_inverted_index.approaches	40
abstract_inverted_index.attributes	121
abstract_inverted_index.confidence	54
abstract_inverted_index.impressive	5
abstract_inverted_index.intraclass	146
abstract_inverted_index.leveraging	11
abstract_inverted_index.particular	87
abstract_inverted_index.PromptAlign	208
abstract_inverted_index.calibration	174, 182
abstract_inverted_index.degradation	52
abstract_inverted_index.effectively	171
abstract_inverted_index.experiments	157
abstract_inverted_index.inter-class	150
abstract_inverted_index.overfitting	84
abstract_inverted_index.performance	6, 18
abstract_inverted_index.recognition	9
abstract_inverted_index.calibration.	55
abstract_inverted_index.demonstrated	4
abstract_inverted_index.(NeurIPS'23).	209
abstract_inverted_index.applicability	59
abstract_inverted_index.applications.	64
abstract_inverted_index.architectures	161
abstract_inverted_index.contributions	68
abstract_inverted_index.Unfortunately,	34
abstract_inverted_index.initialization	79, 109
abstract_inverted_index.miscalibration	96
abstract_inverted_index.regularization	142
abstract_inverted_index.Vision-language	0
abstract_inverted_index.self-supervised	12
abstract_inverted_index.https://github.com/rhebbalaguppe/TCA_PromptWithoutPanic.	216
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	2
citation_normalized_percentile