Is Parameter Collision Hindering Continual Learning in LLMs? Article Swipe

PDF

Shuo Yang , Kun-Peng Ning , Yuyang Liu , Jianfeng Yao , Y. Tian , Yanchen Song , Yuan Li ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2410.10179

Large Language Models (LLMs) often suffer from catastrophic forgetting when learning multiple tasks sequentially, making continual learning (CL) essential for their dynamic deployment. Existing state-of-the-art (SOTA) methods, such as O-LoRA, typically focus on constructing orthogonality tasks to decouple parameter interdependence from various domains.In this paper, we reveal that building non-collision parameters is a more critical factor in addressing CL challenges. Our theoretical and experimental analyses demonstrate that non-collision parameters can provide better task orthogonality, which is a sufficient but unnecessary condition. Furthermore, knowledge from multiple domains will be preserved in non-collision parameter subspaces, making it more difficult to forget previously seen data. Leveraging this insight, we propose Non-collision Low-Rank Adaptation (N-LoRA), a simple yet effective approach leveraging low collision rates to enhance CL in LLMs. Experimental results on multiple CL benchmarks indicate that N-LoRA achieves superior performance (+2.9), higher task orthogonality (*4.1 times), and lower parameter collision (*58.1 times) than SOTA methods.

Related Topics

Economics

Computer Science

Computer Security

Concepts

Collision Economics Computer science Computer security

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2410.10179
PDF: https://arxiv.org/pdf/2410.10179
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4403580697

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4403580697

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2410.10179

Digital Object Identifier
Title: Is Parameter Collision Hindering Continual Learning in LLMs?

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-10-14

Full publication date if available
Authors: Shuo Yang, Kun-Peng Ning, Yuyang Liu, Jianfeng Yao, Y. Tian, Yanchen Song, Yuan Li

List of authors in order
Landing page: https://arxiv.org/abs/2410.10179

Publisher landing page
PDF URL: https://arxiv.org/pdf/2410.10179

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2410.10179

Direct OA link when available
Concepts: Collision, Economics, Computer science, Computer security

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4403580697
doi	https://doi.org/10.48550/arxiv.2410.10179
ids.doi	https://doi.org/10.48550/arxiv.2410.10179
ids.openalex	https://openalex.org/W4403580697
fwci
type	preprint
title	Is Parameter Collision Hindering Continual Learning in LLMs?
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11902
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.8521000146865845
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Intelligent Tutoring Systems and Adaptive Learning
topics[1].id	https://openalex.org/T14470
topics[1].field.id	https://openalex.org/fields/22
topics[1].field.display_name	Engineering
topics[1].score	0.8391000032424927
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/2207
topics[1].subfield.display_name	Control and Systems Engineering
topics[1].display_name	Advanced Data Processing Techniques
topics[2].id	https://openalex.org/T10820
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.8184000253677368
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Fuzzy Logic and Control Systems
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C121704057
concepts[0].level	2
concepts[0].score	0.7931212782859802
concepts[0].wikidata	https://www.wikidata.org/wiki/Q352070
concepts[0].display_name	Collision
concepts[1].id	https://openalex.org/C162324750
concepts[1].level	0
concepts[1].score	0.3501591980457306
concepts[1].wikidata	https://www.wikidata.org/wiki/Q8134
concepts[1].display_name	Economics
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.25083106756210327
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C38652104
concepts[3].level	1
concepts[3].score	0.20118668675422668
concepts[3].wikidata	https://www.wikidata.org/wiki/Q3510521
concepts[3].display_name	Computer security
keywords[0].id	https://openalex.org/keywords/collision
keywords[0].score	0.7931212782859802
keywords[0].display_name	Collision
keywords[1].id	https://openalex.org/keywords/economics
keywords[1].score	0.3501591980457306
keywords[1].display_name	Economics
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.25083106756210327
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/computer-security
keywords[3].score	0.20118668675422668
keywords[3].display_name	Computer security
language	en
locations[0].id	pmh:oai:arXiv.org:2410.10179
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2410.10179
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2410.10179
locations[1].id	doi:10.48550/arxiv.2410.10179
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2410.10179
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5115596917
authorships[0].author.orcid	https://orcid.org/0009-0008-0296-0183
authorships[0].author.display_name	Shuo Yang
authorships[0].author_position	first
authorships[0].raw_author_name	Yang, Shuo
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5114338331
authorships[1].author.orcid
authorships[1].author.display_name	Kun-Peng Ning
authorships[1].author_position	middle
authorships[1].raw_author_name	Ning, Kun-Peng
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5100373459
authorships[2].author.orcid	https://orcid.org/0000-0002-6182-3786
authorships[2].author.display_name	Yuyang Liu
authorships[2].author_position	middle
authorships[2].raw_author_name	Liu, Yu-Yang
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5075620860
authorships[3].author.orcid	https://orcid.org/0000-0001-6356-870X
authorships[3].author.display_name	Jianfeng Yao
authorships[3].author_position	middle
authorships[3].raw_author_name	Yao, Jia-Yu
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5108056266
authorships[4].author.orcid	https://orcid.org/0000-0002-3626-2331
authorships[4].author.display_name	Y. Tian
authorships[4].author_position	middle
authorships[4].raw_author_name	Tian, Yong-Hong
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5009504566
authorships[5].author.orcid	https://orcid.org/0000-0001-9200-9835
authorships[5].author.display_name	Yanchen Song
authorships[5].author_position	middle
authorships[5].raw_author_name	Song, Yi-Bing
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5100621615
authorships[6].author.orcid	https://orcid.org/0000-0002-2117-7425
authorships[6].author.display_name	Yuan Li
authorships[6].author_position	last
authorships[6].raw_author_name	Yuan, Li
authorships[6].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2410.10179
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Is Parameter Collision Hindering Continual Learning in LLMs?
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11902
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.8521000146865845
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Intelligent Tutoring Systems and Adaptive Learning
related_works	https://openalex.org/W4391375266, https://openalex.org/W3113932901, https://openalex.org/W650625605, https://openalex.org/W2801025257, https://openalex.org/W2551285827, https://openalex.org/W1919219501, https://openalex.org/W2027504272, https://openalex.org/W2069703662, https://openalex.org/W2372181157, https://openalex.org/W2759268906
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2410.10179
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2410.10179
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2410.10179
primary_location.id	pmh:oai:arXiv.org:2410.10179
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2410.10179
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2410.10179
publication_date	2024-10-14
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	52, 76, 111
abstract_inverted_index.CL	58, 122, 129
abstract_inverted_index.as	28
abstract_inverted_index.be	87
abstract_inverted_index.in	56, 89, 123
abstract_inverted_index.is	51, 75
abstract_inverted_index.it	94
abstract_inverted_index.on	32, 127
abstract_inverted_index.to	36, 97, 120
abstract_inverted_index.we	45, 105
abstract_inverted_index.Our	60
abstract_inverted_index.and	62, 143
abstract_inverted_index.but	78
abstract_inverted_index.can	69
abstract_inverted_index.for	19
abstract_inverted_index.low	117
abstract_inverted_index.yet	113
abstract_inverted_index.(CL)	17
abstract_inverted_index.SOTA	150
abstract_inverted_index.from	6, 40, 83
abstract_inverted_index.more	53, 95
abstract_inverted_index.seen	100
abstract_inverted_index.such	27
abstract_inverted_index.task	72, 139
abstract_inverted_index.than	149
abstract_inverted_index.that	47, 66, 132
abstract_inverted_index.this	43, 103
abstract_inverted_index.when	9
abstract_inverted_index.will	86
abstract_inverted_index.(*4.1	141
abstract_inverted_index.LLMs.	124
abstract_inverted_index.Large	0
abstract_inverted_index.data.	101
abstract_inverted_index.focus	31
abstract_inverted_index.lower	144
abstract_inverted_index.often	4
abstract_inverted_index.rates	119
abstract_inverted_index.tasks	12, 35
abstract_inverted_index.their	20
abstract_inverted_index.which	74
abstract_inverted_index.(*58.1	147
abstract_inverted_index.(LLMs)	3
abstract_inverted_index.(SOTA)	25
abstract_inverted_index.Models	2
abstract_inverted_index.N-LoRA	133
abstract_inverted_index.better	71
abstract_inverted_index.factor	55
abstract_inverted_index.forget	98
abstract_inverted_index.higher	138
abstract_inverted_index.making	14, 93
abstract_inverted_index.paper,	44
abstract_inverted_index.reveal	46
abstract_inverted_index.simple	112
abstract_inverted_index.suffer	5
abstract_inverted_index.times)	148
abstract_inverted_index.(+2.9),	137
abstract_inverted_index.O-LoRA,	29
abstract_inverted_index.domains	85
abstract_inverted_index.dynamic	21
abstract_inverted_index.enhance	121
abstract_inverted_index.propose	106
abstract_inverted_index.provide	70
abstract_inverted_index.results	126
abstract_inverted_index.times),	142
abstract_inverted_index.various	41
abstract_inverted_index.Existing	23
abstract_inverted_index.Language	1
abstract_inverted_index.Low-Rank	108
abstract_inverted_index.achieves	134
abstract_inverted_index.analyses	64
abstract_inverted_index.approach	115
abstract_inverted_index.building	48
abstract_inverted_index.critical	54
abstract_inverted_index.decouple	37
abstract_inverted_index.indicate	131
abstract_inverted_index.insight,	104
abstract_inverted_index.learning	10, 16
abstract_inverted_index.methods,	26
abstract_inverted_index.methods.	151
abstract_inverted_index.multiple	11, 84, 128
abstract_inverted_index.superior	135
abstract_inverted_index.(N-LoRA),	110
abstract_inverted_index.collision	118, 146
abstract_inverted_index.continual	15
abstract_inverted_index.difficult	96
abstract_inverted_index.effective	114
abstract_inverted_index.essential	18
abstract_inverted_index.knowledge	82
abstract_inverted_index.parameter	38, 91, 145
abstract_inverted_index.preserved	88
abstract_inverted_index.typically	30
abstract_inverted_index.Adaptation	109
abstract_inverted_index.Leveraging	102
abstract_inverted_index.addressing	57
abstract_inverted_index.benchmarks	130
abstract_inverted_index.condition.	80
abstract_inverted_index.domains.In	42
abstract_inverted_index.forgetting	8
abstract_inverted_index.leveraging	116
abstract_inverted_index.parameters	50, 68
abstract_inverted_index.previously	99
abstract_inverted_index.subspaces,	92
abstract_inverted_index.sufficient	77
abstract_inverted_index.challenges.	59
abstract_inverted_index.demonstrate	65
abstract_inverted_index.deployment.	22
abstract_inverted_index.performance	136
abstract_inverted_index.theoretical	61
abstract_inverted_index.unnecessary	79
abstract_inverted_index.Experimental	125
abstract_inverted_index.Furthermore,	81
abstract_inverted_index.catastrophic	7
abstract_inverted_index.constructing	33
abstract_inverted_index.experimental	63
abstract_inverted_index.Non-collision	107
abstract_inverted_index.non-collision	49, 67, 90
abstract_inverted_index.orthogonality	34, 140
abstract_inverted_index.sequentially,	13
abstract_inverted_index.orthogonality,	73
abstract_inverted_index.interdependence	39
abstract_inverted_index.state-of-the-art	24
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	7
citation_normalized_percentile