Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models Article Swipe

PDF

Yae Jee Cho , Luyang Liu , Zheng Xu , Aldi Fahrezi , Gauri Joshi ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2401.06432

Foundation models (FMs) adapt well to specific domains or tasks with fine-tuning, and federated learning (FL) enables the potential for privacy-preserving fine-tuning of the FMs with on-device local data. For federated fine-tuning of FMs, we consider the FMs with small to medium parameter sizes of single digit billion at maximum, referred to as on-device FMs (ODFMs) that can be deployed on devices for inference but can only be fine-tuned with parameter efficient methods. In our work, we tackle the data and system heterogeneity problem of federated fine-tuning of ODFMs by proposing a novel method using heterogeneous low-rank approximations (LoRAs), namely HetLoRA. First, we show that the naive approach of using homogeneous LoRA ranks across devices face a trade-off between overfitting and slow convergence, and thus propose HetLoRA, which allows heterogeneous ranks across client devices and efficiently aggregates and distributes these heterogeneous LoRA modules. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, HetLoRA combines the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, HetLoRA offers enhanced computation efficiency compared to full fine-tuning, making it suitable for federated fine-tuning across heterogeneous devices.

Related Topics

Artificial Intelligence

Concepts

Computer science Overfitting Fine-tuning Homogeneous Pruning Computation Inference Convergence (economics) Rank (graph theory) Distributed computing Computer engineering Artificial intelligence Algorithm Artificial neural network Economics Economic growth Physics Thermodynamics Combinatorics Biology Agronomy Quantum mechanics Mathematics

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2401.06432
PDF: https://arxiv.org/pdf/2401.06432
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4390897567

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4390897567

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2401.06432

Digital Object Identifier
Title: Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-01-12

Full publication date if available
Authors: Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi

List of authors in order
Landing page: https://arxiv.org/abs/2401.06432

Publisher landing page
PDF URL: https://arxiv.org/pdf/2401.06432

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2401.06432

Direct OA link when available
Concepts: Computer science, Overfitting, Fine-tuning, Homogeneous, Pruning, Computation, Inference, Convergence (economics), Rank (graph theory), Distributed computing, Computer engineering, Artificial intelligence, Algorithm, Artificial neural network, Economics, Economic growth, Physics, Thermodynamics, Combinatorics, Biology, Agronomy, Quantum mechanics, Mathematics

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2024: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4390897567
doi	https://doi.org/10.48550/arxiv.2401.06432
ids.doi	https://doi.org/10.48550/arxiv.2401.06432
ids.openalex	https://openalex.org/W4390897567
fwci
type	preprint
title	Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11612
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9969000220298767
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Stochastic Gradient Optimization Techniques
topics[1].id	https://openalex.org/T10764
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9872000217437744
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Privacy-Preserving Technologies in Data
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.810480535030365
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C22019652
concepts[1].level	3
concepts[1].score	0.6439402103424072
concepts[1].wikidata	https://www.wikidata.org/wiki/Q331309
concepts[1].display_name	Overfitting
concepts[2].id	https://openalex.org/C157524613
concepts[2].level	2
concepts[2].score	0.6240371465682983
concepts[2].wikidata	https://www.wikidata.org/wiki/Q2828883
concepts[2].display_name	Fine-tuning
concepts[3].id	https://openalex.org/C66882249
concepts[3].level	2
concepts[3].score	0.5621689558029175
concepts[3].wikidata	https://www.wikidata.org/wiki/Q169336
concepts[3].display_name	Homogeneous
concepts[4].id	https://openalex.org/C108010975
concepts[4].level	2
concepts[4].score	0.547289252281189
concepts[4].wikidata	https://www.wikidata.org/wiki/Q500094
concepts[4].display_name	Pruning
concepts[5].id	https://openalex.org/C45374587
concepts[5].level	2
concepts[5].score	0.5283891558647156
concepts[5].wikidata	https://www.wikidata.org/wiki/Q12525525
concepts[5].display_name	Computation
concepts[6].id	https://openalex.org/C2776214188
concepts[6].level	2
concepts[6].score	0.5249959230422974
concepts[6].wikidata	https://www.wikidata.org/wiki/Q408386
concepts[6].display_name	Inference
concepts[7].id	https://openalex.org/C2777303404
concepts[7].level	2
concepts[7].score	0.4979071617126465
concepts[7].wikidata	https://www.wikidata.org/wiki/Q759757
concepts[7].display_name	Convergence (economics)
concepts[8].id	https://openalex.org/C164226766
concepts[8].level	2
concepts[8].score	0.4729839563369751
concepts[8].wikidata	https://www.wikidata.org/wiki/Q7293202
concepts[8].display_name	Rank (graph theory)
concepts[9].id	https://openalex.org/C120314980
concepts[9].level	1
concepts[9].score	0.4167424142360687
concepts[9].wikidata	https://www.wikidata.org/wiki/Q180634
concepts[9].display_name	Distributed computing
concepts[10].id	https://openalex.org/C113775141
concepts[10].level	1
concepts[10].score	0.329401433467865
concepts[10].wikidata	https://www.wikidata.org/wiki/Q428691
concepts[10].display_name	Computer engineering
concepts[11].id	https://openalex.org/C154945302
concepts[11].level	1
concepts[11].score	0.30458521842956543
concepts[11].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[11].display_name	Artificial intelligence
concepts[12].id	https://openalex.org/C11413529
concepts[12].level	1
concepts[12].score	0.2755963206291199
concepts[12].wikidata	https://www.wikidata.org/wiki/Q8366
concepts[12].display_name	Algorithm
concepts[13].id	https://openalex.org/C50644808
concepts[13].level	2
concepts[13].score	0.1238929033279419
concepts[13].wikidata	https://www.wikidata.org/wiki/Q192776
concepts[13].display_name	Artificial neural network
concepts[14].id	https://openalex.org/C162324750
concepts[14].level	0
concepts[14].score	0.0
concepts[14].wikidata	https://www.wikidata.org/wiki/Q8134
concepts[14].display_name	Economics
concepts[15].id	https://openalex.org/C50522688
concepts[15].level	1
concepts[15].score	0.0
concepts[15].wikidata	https://www.wikidata.org/wiki/Q189833
concepts[15].display_name	Economic growth
concepts[16].id	https://openalex.org/C121332964
concepts[16].level	0
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q413
concepts[16].display_name	Physics
concepts[17].id	https://openalex.org/C97355855
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q11473
concepts[17].display_name	Thermodynamics
concepts[18].id	https://openalex.org/C114614502
concepts[18].level	1
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q76592
concepts[18].display_name	Combinatorics
concepts[19].id	https://openalex.org/C86803240
concepts[19].level	0
concepts[19].score	0.0
concepts[19].wikidata	https://www.wikidata.org/wiki/Q420
concepts[19].display_name	Biology
concepts[20].id	https://openalex.org/C6557445
concepts[20].level	1
concepts[20].score	0.0
concepts[20].wikidata	https://www.wikidata.org/wiki/Q173113
concepts[20].display_name	Agronomy
concepts[21].id	https://openalex.org/C62520636
concepts[21].level	1
concepts[21].score	0.0
concepts[21].wikidata	https://www.wikidata.org/wiki/Q944
concepts[21].display_name	Quantum mechanics
concepts[22].id	https://openalex.org/C33923547
concepts[22].level	0
concepts[22].score	0.0
concepts[22].wikidata	https://www.wikidata.org/wiki/Q395
concepts[22].display_name	Mathematics
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.810480535030365
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/overfitting
keywords[1].score	0.6439402103424072
keywords[1].display_name	Overfitting
keywords[2].id	https://openalex.org/keywords/fine-tuning
keywords[2].score	0.6240371465682983
keywords[2].display_name	Fine-tuning
keywords[3].id	https://openalex.org/keywords/homogeneous
keywords[3].score	0.5621689558029175
keywords[3].display_name	Homogeneous
keywords[4].id	https://openalex.org/keywords/pruning
keywords[4].score	0.547289252281189
keywords[4].display_name	Pruning
keywords[5].id	https://openalex.org/keywords/computation
keywords[5].score	0.5283891558647156
keywords[5].display_name	Computation
keywords[6].id	https://openalex.org/keywords/inference
keywords[6].score	0.5249959230422974
keywords[6].display_name	Inference
keywords[7].id	https://openalex.org/keywords/convergence
keywords[7].score	0.4979071617126465
keywords[7].display_name	Convergence (economics)
keywords[8].id	https://openalex.org/keywords/rank
keywords[8].score	0.4729839563369751
keywords[8].display_name	Rank (graph theory)
keywords[9].id	https://openalex.org/keywords/distributed-computing
keywords[9].score	0.4167424142360687
keywords[9].display_name	Distributed computing
keywords[10].id	https://openalex.org/keywords/computer-engineering
keywords[10].score	0.329401433467865
keywords[10].display_name	Computer engineering
keywords[11].id	https://openalex.org/keywords/artificial-intelligence
keywords[11].score	0.30458521842956543
keywords[11].display_name	Artificial intelligence
keywords[12].id	https://openalex.org/keywords/algorithm
keywords[12].score	0.2755963206291199
keywords[12].display_name	Algorithm
keywords[13].id	https://openalex.org/keywords/artificial-neural-network
keywords[13].score	0.1238929033279419
keywords[13].display_name	Artificial neural network
language	en
locations[0].id	pmh:oai:arXiv.org:2401.06432
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2401.06432
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2401.06432
locations[1].id	doi:10.48550/arxiv.2401.06432
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2401.06432
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5025202888
authorships[0].author.orcid	https://orcid.org/0000-0001-6075-2712
authorships[0].author.display_name	Yae Jee Cho
authorships[0].author_position	first
authorships[0].raw_author_name	Cho, Yae Jee
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5036831783
authorships[1].author.orcid	https://orcid.org/0000-0002-9002-3788
authorships[1].author.display_name	Luyang Liu
authorships[1].author_position	middle
authorships[1].raw_author_name	Liu, Luyang
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5100413782
authorships[2].author.orcid	https://orcid.org/0000-0003-0648-0327
authorships[2].author.display_name	Zheng Xu
authorships[2].author_position	middle
authorships[2].raw_author_name	Xu, Zheng
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5093723171
authorships[3].author.orcid
authorships[3].author.display_name	Aldi Fahrezi
authorships[3].author_position	middle
authorships[3].raw_author_name	Fahrezi, Aldi
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5067441201
authorships[4].author.orcid	https://orcid.org/0000-0002-6372-9697
authorships[4].author.display_name	Gauri Joshi
authorships[4].author_position	last
authorships[4].raw_author_name	Joshi, Gauri
authorships[4].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2401.06432
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-01-16T00:00:00
display_name	Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11612
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9969000220298767
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Stochastic Gradient Optimization Techniques
related_works	https://openalex.org/W4362597605, https://openalex.org/W1574414179, https://openalex.org/W3009056573, https://openalex.org/W4297676672, https://openalex.org/W1585770001, https://openalex.org/W2075873371, https://openalex.org/W4245210885, https://openalex.org/W84383711, https://openalex.org/W4242351093, https://openalex.org/W4302305417
cited_by_count	1
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2401.06432
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2401.06432
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2401.06432
primary_location.id	pmh:oai:arXiv.org:2401.06432
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2401.06432
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2401.06432
publication_date	2024-01-12
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	91, 116
abstract_inverted_index.By	143
abstract_inverted_index.In	73
abstract_inverted_index.as	52
abstract_inverted_index.at	48, 151
abstract_inverted_index.be	58, 67
abstract_inverted_index.by	89
abstract_inverted_index.it	186
abstract_inverted_index.of	22, 32, 44, 84, 87, 108, 158
abstract_inverted_index.on	60
abstract_inverted_index.or	8
abstract_inverted_index.to	5, 40, 51, 172, 182
abstract_inverted_index.we	34, 76, 102
abstract_inverted_index.FMs	24, 37, 54
abstract_inverted_index.For	29
abstract_inverted_index.and	12, 80, 120, 123, 134, 137, 148, 160, 168
abstract_inverted_index.but	64
abstract_inverted_index.can	57, 65
abstract_inverted_index.for	19, 62, 188
abstract_inverted_index.our	74
abstract_inverted_index.the	17, 23, 36, 78, 105, 152, 156
abstract_inverted_index.(FL)	15
abstract_inverted_index.FMs,	33
abstract_inverted_index.LoRA	111, 141
abstract_inverted_index.data	79
abstract_inverted_index.face	115
abstract_inverted_index.full	183
abstract_inverted_index.high	159
abstract_inverted_index.only	66
abstract_inverted_index.rank	145
abstract_inverted_index.show	103
abstract_inverted_index.slow	121
abstract_inverted_index.that	56, 104
abstract_inverted_index.thus	124
abstract_inverted_index.well	4
abstract_inverted_index.with	10, 25, 38, 69
abstract_inverted_index.(FMs)	2
abstract_inverted_index.LoRA.	174
abstract_inverted_index.ODFMs	88
abstract_inverted_index.adapt	3
abstract_inverted_index.data.	28
abstract_inverted_index.digit	46
abstract_inverted_index.final	169
abstract_inverted_index.local	27
abstract_inverted_index.naive	106
abstract_inverted_index.novel	92
abstract_inverted_index.ranks	112, 130
abstract_inverted_index.sizes	43
abstract_inverted_index.small	39
abstract_inverted_index.speed	167
abstract_inverted_index.tasks	9
abstract_inverted_index.these	139
abstract_inverted_index.using	94, 109
abstract_inverted_index.which	127, 163
abstract_inverted_index.work,	75
abstract_inverted_index.First,	101
abstract_inverted_index.LoRAs,	162
abstract_inverted_index.across	113, 131, 191
abstract_inverted_index.allows	128
abstract_inverted_index.client	132
abstract_inverted_index.making	185
abstract_inverted_index.medium	41
abstract_inverted_index.method	93
abstract_inverted_index.models	1
abstract_inverted_index.namely	99
abstract_inverted_index.offers	177
abstract_inverted_index.single	45
abstract_inverted_index.system	81
abstract_inverted_index.tackle	77
abstract_inverted_index.(ODFMs)	55
abstract_inverted_index.HetLoRA	154, 176
abstract_inverted_index.between	118
abstract_inverted_index.billion	47
abstract_inverted_index.devices	61, 114, 133
abstract_inverted_index.domains	7
abstract_inverted_index.enables	16
abstract_inverted_index.locally	147
abstract_inverted_index.problem	83
abstract_inverted_index.propose	125
abstract_inverted_index.server,	153
abstract_inverted_index.(LoRAs),	98
abstract_inverted_index.HetLoRA,	126
abstract_inverted_index.HetLoRA.	100
abstract_inverted_index.achieves	164
abstract_inverted_index.applying	144
abstract_inverted_index.approach	107
abstract_inverted_index.combines	155
abstract_inverted_index.compared	171, 181
abstract_inverted_index.consider	35
abstract_inverted_index.deployed	59
abstract_inverted_index.devices.	193
abstract_inverted_index.enhanced	178
abstract_inverted_index.improved	165
abstract_inverted_index.learning	14
abstract_inverted_index.low-rank	96, 161
abstract_inverted_index.maximum,	49
abstract_inverted_index.methods.	72
abstract_inverted_index.modules.	142
abstract_inverted_index.referred	50
abstract_inverted_index.specific	6
abstract_inverted_index.suitable	187
abstract_inverted_index.efficient	71
abstract_inverted_index.federated	13, 30, 85, 189
abstract_inverted_index.inference	63
abstract_inverted_index.on-device	26, 53
abstract_inverted_index.parameter	42, 70
abstract_inverted_index.potential	18
abstract_inverted_index.proposing	90
abstract_inverted_index.trade-off	117
abstract_inverted_index.Foundation	0
abstract_inverted_index.advantages	157
abstract_inverted_index.aggregates	136
abstract_inverted_index.efficiency	180
abstract_inverted_index.fine-tuned	68
abstract_inverted_index.aggregation	150
abstract_inverted_index.computation	179
abstract_inverted_index.convergence	166
abstract_inverted_index.distributes	138
abstract_inverted_index.efficiently	135
abstract_inverted_index.fine-tuning	21, 31, 86, 190
abstract_inverted_index.homogeneous	110, 173
abstract_inverted_index.overfitting	119
abstract_inverted_index.performance	170
abstract_inverted_index.Furthermore,	175
abstract_inverted_index.convergence,	122
abstract_inverted_index.fine-tuning,	11, 184
abstract_inverted_index.self-pruning	146
abstract_inverted_index.heterogeneity	82
abstract_inverted_index.heterogeneous	95, 129, 140, 192
abstract_inverted_index.approximations	97
abstract_inverted_index.sparsity-weighted	149
abstract_inverted_index.privacy-preserving	20
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile