Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning Article Swipe

PDF

Jingchen Sun , Rajni Sharma , Vishnu Suresh Lokhande , Changyou Chen ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2407.15894

Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However, recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper, we propose a Cross-modal Aligned Feature Tuning (Craft) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness, we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6.1\%$ in the Base-to-Novel generalization task, $5.8\%$ in the group robustness task, and $2.7\%$ in the out-of-distribution tasks. The code will be available at https://github.com/Jingchensun/Craft

Related Topics

Concepts

Craft Modal Robustness (evolution) Computer science Materials science Art Visual arts Composite material Biochemistry Chemistry Gene

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2407.15894
PDF: https://arxiv.org/pdf/2407.15894
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4406073449

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4406073449

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2407.15894

Digital Object Identifier
Title: Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-07-22

Full publication date if available
Authors: Jingchen Sun, Rajni Sharma, Vishnu Suresh Lokhande, Changyou Chen

List of authors in order
Landing page: https://arxiv.org/abs/2407.15894

Publisher landing page
PDF URL: https://arxiv.org/pdf/2407.15894

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2407.15894

Direct OA link when available
Concepts: Craft, Modal, Robustness (evolution), Computer science, Materials science, Art, Visual arts, Composite material, Biochemistry, Chemistry, Gene

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4406073449
doi	https://doi.org/10.48550/arxiv.2407.15894
ids.doi	https://doi.org/10.48550/arxiv.2407.15894
ids.openalex	https://openalex.org/W4406073449
fwci	0.0
type	preprint
title	Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11159
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.9505000114440918
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2209
topics[0].subfield.display_name	Industrial and Manufacturing Engineering
topics[0].display_name	Manufacturing Process and Optimization
topics[1].id	https://openalex.org/T10653
topics[1].field.id	https://openalex.org/fields/22
topics[1].field.display_name	Engineering
topics[1].score	0.926800012588501
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/2207
topics[1].subfield.display_name	Control and Systems Engineering
topics[1].display_name	Robot Manipulation and Learning
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2779732396
concepts[0].level	2
concepts[0].score	0.8636564016342163
concepts[0].wikidata	https://www.wikidata.org/wiki/Q2207288
concepts[0].display_name	Craft
concepts[1].id	https://openalex.org/C71139939
concepts[1].level	2
concepts[1].score	0.7743116617202759
concepts[1].wikidata	https://www.wikidata.org/wiki/Q910194
concepts[1].display_name	Modal
concepts[2].id	https://openalex.org/C63479239
concepts[2].level	3
concepts[2].score	0.738836944103241
concepts[2].wikidata	https://www.wikidata.org/wiki/Q7353546
concepts[2].display_name	Robustness (evolution)
concepts[3].id	https://openalex.org/C41008148
concepts[3].level	0
concepts[3].score	0.42830345034599304
concepts[3].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[3].display_name	Computer science
concepts[4].id	https://openalex.org/C192562407
concepts[4].level	0
concepts[4].score	0.2680032551288605
concepts[4].wikidata	https://www.wikidata.org/wiki/Q228736
concepts[4].display_name	Materials science
concepts[5].id	https://openalex.org/C142362112
concepts[5].level	0
concepts[5].score	0.21954545378684998
concepts[5].wikidata	https://www.wikidata.org/wiki/Q735
concepts[5].display_name	Art
concepts[6].id	https://openalex.org/C153349607
concepts[6].level	1
concepts[6].score	0.1511649191379547
concepts[6].wikidata	https://www.wikidata.org/wiki/Q36649
concepts[6].display_name	Visual arts
concepts[7].id	https://openalex.org/C159985019
concepts[7].level	1
concepts[7].score	0.11569583415985107
concepts[7].wikidata	https://www.wikidata.org/wiki/Q181790
concepts[7].display_name	Composite material
concepts[8].id	https://openalex.org/C55493867
concepts[8].level	1
concepts[8].score	0.0
concepts[8].wikidata	https://www.wikidata.org/wiki/Q7094
concepts[8].display_name	Biochemistry
concepts[9].id	https://openalex.org/C185592680
concepts[9].level	0
concepts[9].score	0.0
concepts[9].wikidata	https://www.wikidata.org/wiki/Q2329
concepts[9].display_name	Chemistry
concepts[10].id	https://openalex.org/C104317684
concepts[10].level	2
concepts[10].score	0.0
concepts[10].wikidata	https://www.wikidata.org/wiki/Q7187
concepts[10].display_name	Gene
keywords[0].id	https://openalex.org/keywords/craft
keywords[0].score	0.8636564016342163
keywords[0].display_name	Craft
keywords[1].id	https://openalex.org/keywords/modal
keywords[1].score	0.7743116617202759
keywords[1].display_name	Modal
keywords[2].id	https://openalex.org/keywords/robustness
keywords[2].score	0.738836944103241
keywords[2].display_name	Robustness (evolution)
keywords[3].id	https://openalex.org/keywords/computer-science
keywords[3].score	0.42830345034599304
keywords[3].display_name	Computer science
keywords[4].id	https://openalex.org/keywords/materials-science
keywords[4].score	0.2680032551288605
keywords[4].display_name	Materials science
keywords[5].id	https://openalex.org/keywords/art
keywords[5].score	0.21954545378684998
keywords[5].display_name	Art
keywords[6].id	https://openalex.org/keywords/visual-arts
keywords[6].score	0.1511649191379547
keywords[6].display_name	Visual arts
keywords[7].id	https://openalex.org/keywords/composite-material
keywords[7].score	0.11569583415985107
keywords[7].display_name	Composite material
language	en
locations[0].id	pmh:oai:arXiv.org:2407.15894
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2407.15894
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2407.15894
locations[1].id	doi:10.48550/arxiv.2407.15894
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2407.15894
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5110936166
authorships[0].author.orcid
authorships[0].author.display_name	Jingchen Sun
authorships[0].author_position	first
authorships[0].raw_author_name	Sun, Jingchen
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5102912940
authorships[1].author.orcid	https://orcid.org/0000-0003-3556-1660
authorships[1].author.display_name	Rajni Sharma
authorships[1].author_position	middle
authorships[1].raw_author_name	Sharma, Rohan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5081203502
authorships[2].author.orcid	https://orcid.org/0000-0002-4354-0125
authorships[2].author.display_name	Vishnu Suresh Lokhande
authorships[2].author_position	middle
authorships[2].raw_author_name	Lokhande, Vishnu Suresh
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5114686221
authorships[3].author.orcid	https://orcid.org/0000-0002-9877-8448
authorships[3].author.display_name	Changyou Chen
authorships[3].author_position	last
authorships[3].raw_author_name	Chen, Changyou
authorships[3].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2407.15894
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11159
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.9505000114440918
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2209
primary_topic.subfield.display_name	Industrial and Manufacturing Engineering
primary_topic.display_name	Manufacturing Process and Optimization
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2912168137, https://openalex.org/W2771752657, https://openalex.org/W2936836059, https://openalex.org/W2920882006, https://openalex.org/W1567987063, https://openalex.org/W2030429945, https://openalex.org/W2391860589
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2407.15894
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2407.15894
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2407.15894
primary_location.id	pmh:oai:arXiv.org:2407.15894
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2407.15894
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2407.15894
publication_date	2024-07-22
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	5, 39, 75, 86
abstract_inverted_index.In	34
abstract_inverted_index.To	103
abstract_inverted_index.as	4
abstract_inverted_index.at	169
abstract_inverted_index.be	167
abstract_inverted_index.by	54
abstract_inverted_index.in	93, 147, 153, 160
abstract_inverted_index.is	52
abstract_inverted_index.of	66, 138, 143
abstract_inverted_index.on	100, 128
abstract_inverted_index.to	13, 27, 30, 46, 122, 145
abstract_inverted_index.up	144
abstract_inverted_index.we	37, 110
abstract_inverted_index.The	126, 164
abstract_inverted_index.and	62, 82, 158
abstract_inverted_index.due	29
abstract_inverted_index.for	9, 69, 74
abstract_inverted_index.has	2
abstract_inverted_index.our	139
abstract_inverted_index.the	59, 67, 70, 106, 118, 136, 148, 154, 161
abstract_inverted_index.Mean	114
abstract_inverted_index.also	96
abstract_inverted_index.code	165
abstract_inverted_index.four	129
abstract_inverted_index.from	58
abstract_inverted_index.lead	26
abstract_inverted_index.loss	78
abstract_inverted_index.more	87
abstract_inverted_index.over	79, 117
abstract_inverted_index.text	81
abstract_inverted_index.that	21
abstract_inverted_index.this	35, 48
abstract_inverted_index.will	166
abstract_inverted_index.with	141
abstract_inverted_index.(MMD)	116
abstract_inverted_index.first	55
abstract_inverted_index.group	155
abstract_inverted_index.image	83
abstract_inverted_index.model	98
abstract_inverted_index.often	25
abstract_inverted_index.shows	135
abstract_inverted_index.task,	151, 157
abstract_inverted_index.Prompt	0
abstract_inverted_index.Tuning	1, 43
abstract_inverted_index.common	90
abstract_inverted_index.domain	61, 124
abstract_inverted_index.issue.	49
abstract_inverted_index.method	45
abstract_inverted_index.models	12
abstract_inverted_index.paper,	36
abstract_inverted_index.prompt	22, 94, 107, 131
abstract_inverted_index.recent	18
abstract_inverted_index.shift.	125
abstract_inverted_index.space.	91
abstract_inverted_index.spaces	121
abstract_inverted_index.tasks.	16, 163
abstract_inverted_index.tuning	23, 95, 132
abstract_inverted_index.$2.7\%$	159
abstract_inverted_index.$5.8\%$	152
abstract_inverted_index.$6.1\%$	146
abstract_inverted_index.(Craft)	44
abstract_inverted_index.Aligned	41
abstract_inverted_index.Feature	42
abstract_inverted_index.Maximum	113
abstract_inverted_index.address	47
abstract_inverted_index.anchors	57
abstract_inverted_index.creates	85
abstract_inverted_index.emerged	3
abstract_inverted_index.feature	76, 120
abstract_inverted_index.further	104
abstract_inverted_index.improve	105
abstract_inverted_index.limited	31
abstract_inverted_index.method,	140
abstract_inverted_index.methods	24
abstract_inverted_index.model's	108
abstract_inverted_index.propose	38, 111
abstract_inverted_index.unified	88
abstract_inverted_index.various	14
abstract_inverted_index.However,	17
abstract_inverted_index.adapting	10
abstract_inverted_index.anchors.	72
abstract_inverted_index.deriving	63
abstract_inverted_index.mitigate	123
abstract_inverted_index.paradigm	8
abstract_inverted_index.relative	64
abstract_inverted_index.research	7, 19
abstract_inverted_index.samples.	33, 102
abstract_inverted_index.selected	71
abstract_inverted_index.training	32
abstract_inverted_index.alignment	51, 77
abstract_inverted_index.available	168
abstract_inverted_index.conducted	53
abstract_inverted_index.different	130
abstract_inverted_index.increases	142
abstract_inverted_index.indicates	20
abstract_inverted_index.prominent	6
abstract_inverted_index.selecting	56
abstract_inverted_index.Optimizing	73
abstract_inverted_index.downstream	15
abstract_inverted_index.embeddings	68
abstract_inverted_index.experiment	127
abstract_inverted_index.minimizing	112
abstract_inverted_index.modalities	84
abstract_inverted_index.robustness	156
abstract_inverted_index.structures	133
abstract_inverted_index.text-image	89
abstract_inverted_index.Cross-modal	40, 50
abstract_inverted_index.Discrepancy	115
abstract_inverted_index.Overfitting	92
abstract_inverted_index.alternative	60
abstract_inverted_index.improvement	137
abstract_inverted_index.overfitting	28
abstract_inverted_index.performance	99
abstract_inverted_index.robustness,	109
abstract_inverted_index.consistently	134
abstract_inverted_index.deteriorates	97
abstract_inverted_index.Base-to-Novel	149
abstract_inverted_index.anchor-aligned	80, 119
abstract_inverted_index.generalization	150
abstract_inverted_index.representations	65
abstract_inverted_index.vision-language	11
abstract_inverted_index.out-of-distribution	101, 162
abstract_inverted_index.https://github.com/Jingchensun/Craft	170
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile