Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout Article Swipe
Andi Zhang
,
X. X. Ding
,
Haofan Wang
,
Steven McDonagh
,
Samuel Kaski
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.03262
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.03262
We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity. Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style. When multiple LoRA modules are merged, for example to generate an object in a particular style, their outputs (semantic vectors) may interfere with each other. Our method guarantees that merged LoRA modules remain orthogonal and thus free from direct interference. However, empirical analysis reveals that such orthogonality does not lead to the semantic disentanglement highlighted in prior work on compositional adaptation. This finding suggests that inter-LoRA orthogonality alone may be insufficient for achieving true semantic compositionality, prompting a re-examination of its role in adapter merging.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2510.03262
- https://arxiv.org/pdf/2510.03262
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4414968593
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4414968593Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2510.03262Digital Object Identifier
- Title
-
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo DropoutWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-09-26Full publication date if available
- Authors
-
Andi Zhang, X. X. Ding, Haofan Wang, Steven McDonagh, Samuel KaskiList of authors in order
- Landing page
-
https://arxiv.org/abs/2510.03262Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2510.03262Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2510.03262Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4414968593 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2510.03262 |
| ids.doi | https://doi.org/10.48550/arxiv.2510.03262 |
| ids.openalex | https://openalex.org/W4414968593 |
| fwci | |
| type | preprint |
| title | Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11195 |
| topics[0].field.id | https://openalex.org/fields/18 |
| topics[0].field.display_name | Decision Sciences |
| topics[0].score | 0.4212000072002411 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1803 |
| topics[0].subfield.display_name | Management Science and Operations Research |
| topics[0].display_name | Simulation Techniques and Applications |
| topics[1].id | https://openalex.org/T11159 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.3837999999523163 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2209 |
| topics[1].subfield.display_name | Industrial and Manufacturing Engineering |
| topics[1].display_name | Manufacturing Process and Optimization |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2510.03262 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2510.03262 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2510.03262 |
| locations[1].id | doi:10.48550/arxiv.2510.03262 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2510.03262 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5077911588 |
| authorships[0].author.orcid | https://orcid.org/0009-0007-4855-5442 |
| authorships[0].author.display_name | Andi Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Andi |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5074531806 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-2807-2597 |
| authorships[1].author.display_name | X. X. Ding |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ding, Xuan |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5101497359 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-2098-453X |
| authorships[2].author.display_name | Haofan Wang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Wang, Haofan |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5052824649 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-7025-5197 |
| authorships[3].author.display_name | Steven McDonagh |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | McDonagh, Steven |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5018305257 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-1925-9154 |
| authorships[4].author.display_name | Samuel Kaski |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Kaski, Samuel |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2510.03262 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-09T00:00:00 |
| display_name | Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11195 |
| primary_topic.field.id | https://openalex.org/fields/18 |
| primary_topic.field.display_name | Decision Sciences |
| primary_topic.score | 0.4212000072002411 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1803 |
| primary_topic.subfield.display_name | Management Science and Operations Research |
| primary_topic.display_name | Simulation Techniques and Applications |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2510.03262 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2510.03262 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2510.03262 |
| primary_location.id | pmh:oai:arXiv.org:2510.03262 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2510.03262 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2510.03262 |
| publication_date | 2025-09-26 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 6, 24, 33, 37, 45, 60, 124 |
| abstract_inverted_index.We | 0 |
| abstract_inverted_index.an | 42, 57 |
| abstract_inverted_index.as | 41 |
| abstract_inverted_index.be | 116 |
| abstract_inverted_index.in | 59, 102, 129 |
| abstract_inverted_index.of | 126 |
| abstract_inverted_index.on | 105 |
| abstract_inverted_index.or | 44 |
| abstract_inverted_index.to | 35, 55, 97 |
| abstract_inverted_index.Our | 72 |
| abstract_inverted_index.and | 81 |
| abstract_inverted_index.are | 51 |
| abstract_inverted_index.for | 28, 53, 118 |
| abstract_inverted_index.its | 127 |
| abstract_inverted_index.may | 67, 115 |
| abstract_inverted_index.not | 95 |
| abstract_inverted_index.the | 98 |
| abstract_inverted_index.LoRA | 49, 77 |
| abstract_inverted_index.This | 108 |
| abstract_inverted_index.When | 47 |
| abstract_inverted_index.does | 94 |
| abstract_inverted_index.each | 70 |
| abstract_inverted_index.free | 83 |
| abstract_inverted_index.from | 84 |
| abstract_inverted_index.lead | 96 |
| abstract_inverted_index.role | 128 |
| abstract_inverted_index.such | 40, 92 |
| abstract_inverted_index.that | 8, 75, 91, 111 |
| abstract_inverted_index.thus | 82 |
| abstract_inverted_index.time | 19 |
| abstract_inverted_index.true | 120 |
| abstract_inverted_index.when | 12 |
| abstract_inverted_index.with | 69 |
| abstract_inverted_index.work | 104 |
| abstract_inverted_index.Carlo | 4 |
| abstract_inverted_index.Monte | 3 |
| abstract_inverted_index.alone | 114 |
| abstract_inverted_index.extra | 18 |
| abstract_inverted_index.large | 29 |
| abstract_inverted_index.prior | 103 |
| abstract_inverted_index.their | 63 |
| abstract_inverted_index.direct | 85 |
| abstract_inverted_index.merged | 76 |
| abstract_inverted_index.method | 27, 73 |
| abstract_inverted_index.module | 34 |
| abstract_inverted_index.object | 43, 58 |
| abstract_inverted_index.other. | 71 |
| abstract_inverted_index.remain | 79 |
| abstract_inverted_index.sparse | 14 |
| abstract_inverted_index.strict | 10 |
| abstract_inverted_index.style, | 62 |
| abstract_inverted_index.style. | 46 |
| abstract_inverted_index.trains | 32 |
| abstract_inverted_index.(LoRA), | 23 |
| abstract_inverted_index.adapter | 130 |
| abstract_inverted_index.concept | 39 |
| abstract_inverted_index.example | 54 |
| abstract_inverted_index.finding | 109 |
| abstract_inverted_index.merged, | 52 |
| abstract_inverted_index.models, | 30 |
| abstract_inverted_index.modules | 50, 78 |
| abstract_inverted_index.outputs | 64 |
| abstract_inverted_index.popular | 25 |
| abstract_inverted_index.propose | 1 |
| abstract_inverted_index.reveals | 90 |
| abstract_inverted_index.vectors | 16 |
| abstract_inverted_index.without | 17 |
| abstract_inverted_index.Dropout, | 5 |
| abstract_inverted_index.However, | 87 |
| abstract_inverted_index.Low-Rank | 21 |
| abstract_inverted_index.analysis | 89 |
| abstract_inverted_index.enforces | 9 |
| abstract_inverted_index.generate | 56 |
| abstract_inverted_index.merging. | 131 |
| abstract_inverted_index.multiple | 48 |
| abstract_inverted_index.semantic | 15, 99, 121 |
| abstract_inverted_index.specific | 38 |
| abstract_inverted_index.suggests | 110 |
| abstract_inverted_index.vectors) | 66 |
| abstract_inverted_index.(semantic | 65 |
| abstract_inverted_index.achieving | 119 |
| abstract_inverted_index.combining | 13 |
| abstract_inverted_index.empirical | 88 |
| abstract_inverted_index.interfere | 68 |
| abstract_inverted_index.mechanism | 7 |
| abstract_inverted_index.prompting | 123 |
| abstract_inverted_index.represent | 36 |
| abstract_inverted_index.typically | 31 |
| abstract_inverted_index.Adaptation | 22 |
| abstract_inverted_index.Orthogonal | 2 |
| abstract_inverted_index.guarantees | 74 |
| abstract_inverted_index.inter-LoRA | 112 |
| abstract_inverted_index.orthogonal | 80 |
| abstract_inverted_index.particular | 61 |
| abstract_inverted_index.adaptation. | 107 |
| abstract_inverted_index.complexity. | 20 |
| abstract_inverted_index.fine-tuning | 26 |
| abstract_inverted_index.highlighted | 101 |
| abstract_inverted_index.insufficient | 117 |
| abstract_inverted_index.compositional | 106 |
| abstract_inverted_index.interference. | 86 |
| abstract_inverted_index.orthogonality | 11, 93, 113 |
| abstract_inverted_index.re-examination | 125 |
| abstract_inverted_index.disentanglement | 100 |
| abstract_inverted_index.compositionality, | 122 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |