Sparse Orthogonal Parameters Tuning for Continual Learning Article Swipe

PDF

Kun-Peng Ning , Huai-Jhan Ke , Yuyang Liu , Jiayu Yao , Yonghong Tian , Yuan Li ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2411.02813

Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. These methods typically refrain from updating the pre-trained parameters and instead employ additional adapters, prompts, and classifiers. In this paper, we from a novel perspective investigate the benefit of sparse orthogonal parameters for continual learning. We found that merging sparse orthogonality of models learned from multiple streaming tasks has great potential in addressing catastrophic forgetting. Leveraging this insight, we propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning). We hypothesize that the effectiveness of SoTU lies in the transformation of knowledge learned from multiple domains into the fusion of orthogonal delta parameters. Experimental evaluations on diverse CL benchmarks demonstrate the effectiveness of the proposed approach. Notably, SoTU achieves optimal feature representation for streaming data without necessitating complex classifier designs, making it a Plug-and-Play solution.

Related Topics

Computer Science

Artificial Intelligence

Materials Science

Concepts

Computer science Artificial intelligence Materials science

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2411.02813
PDF: https://arxiv.org/pdf/2411.02813
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4404403735

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4404403735

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2411.02813

Digital Object Identifier
Title: Sparse Orthogonal Parameters Tuning for Continual Learning

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-11-05

Full publication date if available
Authors: Kun-Peng Ning, Huai-Jhan Ke, Yuyang Liu, Jiayu Yao, Yonghong Tian, Yuan Li

List of authors in order
Landing page: https://arxiv.org/abs/2411.02813

Publisher landing page
PDF URL: https://arxiv.org/pdf/2411.02813

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2411.02813

Direct OA link when available
Concepts: Computer science, Artificial intelligence, Materials science

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4404403735
doi	https://doi.org/10.48550/arxiv.2411.02813
ids.doi	https://doi.org/10.48550/arxiv.2411.02813
ids.openalex	https://openalex.org/W4404403735
fwci
type	preprint
title	Sparse Orthogonal Parameters Tuning for Continual Learning
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11609
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.9689000248908997
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2212
topics[0].subfield.display_name	Ocean Engineering
topics[0].display_name	Geophysical Methods and Applications
topics[1].id	https://openalex.org/T10860
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9458000063896179
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1711
topics[1].subfield.display_name	Signal Processing
topics[1].display_name	Speech and Audio Processing
topics[2].id	https://openalex.org/T11038
topics[2].field.id	https://openalex.org/fields/22
topics[2].field.display_name	Engineering
topics[2].score	0.9118000268936157
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/2202
topics[2].subfield.display_name	Aerospace Engineering
topics[2].display_name	Advanced SAR Imaging Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.48182931542396545
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C154945302
concepts[1].level	1
concepts[1].score	0.42174673080444336
concepts[1].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[1].display_name	Artificial intelligence
concepts[2].id	https://openalex.org/C192562407
concepts[2].level	0
concepts[2].score	0.32536208629608154
concepts[2].wikidata	https://www.wikidata.org/wiki/Q228736
concepts[2].display_name	Materials science
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.48182931542396545
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/artificial-intelligence
keywords[1].score	0.42174673080444336
keywords[1].display_name	Artificial intelligence
keywords[2].id	https://openalex.org/keywords/materials-science
keywords[2].score	0.32536208629608154
keywords[2].display_name	Materials science
language	en
locations[0].id	pmh:oai:arXiv.org:2411.02813
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2411.02813
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2411.02813
locations[1].id	doi:10.48550/arxiv.2411.02813
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2411.02813
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5056432762
authorships[0].author.orcid	https://orcid.org/0000-0002-8249-9911
authorships[0].author.display_name	Kun-Peng Ning
authorships[0].author_position	first
authorships[0].raw_author_name	Ning, Kun-Peng
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5108901026
authorships[1].author.orcid
authorships[1].author.display_name	Huai-Jhan Ke
authorships[1].author_position	middle
authorships[1].raw_author_name	Ke, Hai-Jian
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5100373457
authorships[2].author.orcid	https://orcid.org/0000-0002-0808-0047
authorships[2].author.display_name	Yuyang Liu
authorships[2].author_position	middle
authorships[2].raw_author_name	Liu, Yu-Yang
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5101756521
authorships[3].author.orcid	https://orcid.org/0000-0001-6242-9388
authorships[3].author.display_name	Jiayu Yao
authorships[3].author_position	middle
authorships[3].raw_author_name	Yao, Jia-Yu
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5100813706
authorships[4].author.orcid
authorships[4].author.display_name	Yonghong Tian
authorships[4].author_position	middle
authorships[4].raw_author_name	Tian, Yong-Hong
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5100409827
authorships[5].author.orcid	https://orcid.org/0000-0003-3600-3118
authorships[5].author.display_name	Yuan Li
authorships[5].author_position	last
authorships[5].raw_author_name	Yuan, Li
authorships[5].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2411.02813
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-11-16T00:00:00
display_name	Sparse Orthogonal Parameters Tuning for Continual Learning
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11609
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.9689000248908997
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2212
primary_topic.subfield.display_name	Ocean Engineering
primary_topic.display_name	Geophysical Methods and Applications
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2016187641, https://openalex.org/W2805339068, https://openalex.org/W4246450666, https://openalex.org/W4388998267, https://openalex.org/W2898370298, https://openalex.org/W2137437058, https://openalex.org/W4390401159
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2411.02813
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2411.02813
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2411.02813
primary_location.id	pmh:oai:arXiv.org:2411.02813
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2411.02813
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2411.02813
publication_date	2024-11-05
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	43, 81, 145
abstract_inverted_index.CL	120
abstract_inverted_index.In	38
abstract_inverted_index.We	56, 92
abstract_inverted_index.in	72, 100
abstract_inverted_index.it	144
abstract_inverted_index.of	49, 62, 97, 103, 112, 125
abstract_inverted_index.on	4, 118
abstract_inverted_index.to	14
abstract_inverted_index.we	41, 79
abstract_inverted_index.and	30, 36
abstract_inverted_index.for	53, 135
abstract_inverted_index.has	69
abstract_inverted_index.the	27, 47, 95, 101, 110, 123, 126
abstract_inverted_index.yet	83
abstract_inverted_index.SoTU	87, 98, 130
abstract_inverted_index.data	137
abstract_inverted_index.from	25, 42, 65, 106
abstract_inverted_index.have	8
abstract_inverted_index.into	109
abstract_inverted_index.lies	99
abstract_inverted_index.that	58, 94
abstract_inverted_index.this	39, 77
abstract_inverted_index.(PTM)	7
abstract_inverted_index.These	21
abstract_inverted_index.adapt	13
abstract_inverted_index.based	3
abstract_inverted_index.delta	114
abstract_inverted_index.found	57
abstract_inverted_index.great	70
abstract_inverted_index.novel	44, 82
abstract_inverted_index.tasks	17, 68
abstract_inverted_index.which	12
abstract_inverted_index.called	86
abstract_inverted_index.employ	32
abstract_inverted_index.fusion	111
abstract_inverted_index.gained	10
abstract_inverted_index.making	143
abstract_inverted_index.method	85
abstract_inverted_index.models	6, 63
abstract_inverted_index.paper,	40
abstract_inverted_index.sparse	50, 60
abstract_inverted_index.(Sparse	88
abstract_inverted_index.benefit	48
abstract_inverted_index.complex	140
abstract_inverted_index.diverse	119
abstract_inverted_index.domains	108
abstract_inverted_index.feature	133
abstract_inverted_index.instead	31
abstract_inverted_index.learned	64, 105
abstract_inverted_index.merging	59
abstract_inverted_index.methods	2, 22
abstract_inverted_index.optimal	132
abstract_inverted_index.propose	80
abstract_inverted_index.refrain	24
abstract_inverted_index.without	18, 138
abstract_inverted_index.Notably,	129
abstract_inverted_index.TUning).	91
abstract_inverted_index.achieves	131
abstract_inverted_index.designs,	142
abstract_inverted_index.insight,	78
abstract_inverted_index.learning	1
abstract_inverted_index.multiple	66, 107
abstract_inverted_index.prompts,	35
abstract_inverted_index.proposed	127
abstract_inverted_index.recently	9
abstract_inverted_index.updating	26
abstract_inverted_index.Continual	0
abstract_inverted_index.adapters,	34
abstract_inverted_index.approach.	128
abstract_inverted_index.attention	11
abstract_inverted_index.continual	54
abstract_inverted_index.effective	84
abstract_inverted_index.knowledge	104
abstract_inverted_index.learning.	55
abstract_inverted_index.potential	71
abstract_inverted_index.solution.	147
abstract_inverted_index.streaming	67, 136
abstract_inverted_index.typically	23
abstract_inverted_index.Leveraging	76
abstract_inverted_index.Orthogonal	89
abstract_inverted_index.Parameters	90
abstract_inverted_index.additional	33
abstract_inverted_index.addressing	73
abstract_inverted_index.benchmarks	121
abstract_inverted_index.classifier	141
abstract_inverted_index.downstream	16
abstract_inverted_index.orthogonal	51, 113
abstract_inverted_index.parameters	29, 52
abstract_inverted_index.successive	15
abstract_inverted_index.demonstrate	122
abstract_inverted_index.evaluations	117
abstract_inverted_index.forgetting.	20, 75
abstract_inverted_index.hypothesize	93
abstract_inverted_index.investigate	46
abstract_inverted_index.parameters.	115
abstract_inverted_index.perspective	45
abstract_inverted_index.pre-trained	5, 28
abstract_inverted_index.Experimental	116
abstract_inverted_index.catastrophic	19, 74
abstract_inverted_index.classifiers.	37
abstract_inverted_index.Plug-and-Play	146
abstract_inverted_index.effectiveness	96, 124
abstract_inverted_index.necessitating	139
abstract_inverted_index.orthogonality	61
abstract_inverted_index.representation	134
abstract_inverted_index.transformation	102
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile